Data teams have spent years building pipelines that copy operational data into warehouses, lakes, customer platforms, and reporting stores. That pattern still makes sense for many workloads. The mistake is treating copying as the default even when the consumer only needs governed read access to data that already lives in a capable system.
Zero-copy data access changes that starting point. Instead of moving a full dataset into another platform before it can be used, the consuming system reads the source through a shared table, external table, federated query, storage pointer, or similar mechanism. Salesforce’s current zero-copy data federation overview describes both query federation and file federation, which reflects a broader shift across cloud data platforms toward access without routine replication. The useful question isn’t whether zero copy is modern. The useful question is whether a specific workload needs another persistent copy at all.
That distinction matters because ingestion creates obligations. A copied dataset needs a pipeline, a refresh policy, access controls, retention rules, monitoring, failure recovery, and a plan for schema changes. It can also create a second version of the same record that becomes stale between loads. Zero-copy data access avoids some of that work, but it adds its own dependencies on remote compute, network paths, source availability, query pushdown, and shared governance. The right design comes from comparing those tradeoffs against the workload, not from choosing a fashionable architecture.
TL;DR
Zero-copy data access is a strong fit when another platform needs current, mostly read-only access to data that already sits in a well-governed warehouse or lake. It can cut duplicate storage, remove sync delays, and reduce the number of copies that security teams have to track. It works best when queries can be filtered near the source and when the source can handle the extra read load.
• Leave data in place when freshness matters more than local query speed and the source can serve the workload safely.
• Prefer federation for low-frequency analysis, cross-platform discovery, and large datasets where copying most rows would add little value.
• Ingest hot data when repeated queries, strict latency targets, source limits, or heavy transformations make remote access costly or fragile.
• Treat zero copy as an access pattern, not a promise that no bytes ever cross a network.
• Measure source compute, network transfer, query latency, cache behavior, and operational effort before deciding which tables stay remote.
What Zero-Copy Data Access Actually Means
Zero-copy data access means a consuming platform can use data without first creating and maintaining a full persistent replica inside its own storage layer. The source remains authoritative. The consumer usually stores metadata such as table definitions, permissions, object references, or catalog entries, then reads the underlying data when a query or application needs it. The exact mechanism varies by platform, so the phrase should describe the architectural result rather than one technical method.
This is why zero-copy data integration can take several forms. A platform may send SQL to a remote warehouse and return only the query result. It may read Parquet or Iceberg files directly from object storage. It may expose a secure share that another engine can query. It may create a logical shortcut that makes an external folder appear local. Salesforce’s technical explanation of zero copy uses virtual table metadata to describe one such model, while other platforms expose similar ideas through external catalogs and storage references. In each case, the data remains under the source system’s storage control until a query reads it.
The phrase can also be misunderstood. This approach doesn’t mean zero network traffic, zero compute, or zero temporary state. A federated query can still scan source files, run source-side compute, move result sets, spill intermediate data, or create caches. Some services may materialize selected results for performance or downstream features. The meaningful difference is that the architecture doesn’t require a recurring pipeline that copies the entire source dataset into another managed store before it becomes usable.
That makes in-place access different from zero-copy cloning inside one database. A clone may avoid duplicating physical storage blocks at creation time while still behaving like a local database object. The focus here is cross-system access: deciding when analytics, applications, AI workflows, or customer platforms can work against data where it already lives rather than making another long-lived copy first.
How In-Place Access Works
Most in-place designs rely on query federation, storage-level access, or a secure sharing layer. Query federation sends some or all of a query to the remote system. The remote engine scans its own data and returns a result to the consumer. AWS documents this model in Athena Federated Query, including predicate pushdown that reduces the rows moved back to Athena. Good pushdown is important because remote access becomes much less attractive when the consumer pulls a large raw result across the network and filters it only after transfer.
Query Federation
With query federation, both systems can take part in execution. The consumer parses the request, determines what can run remotely, sends work to the source, then finishes any remaining joins or calculations. Databricks describes query federation as a way to query external systems in place for use cases such as on-demand reporting, proof-of-concept work, and incremental migration. This pattern is often useful when the source already has an efficient query engine and the consumer needs access more than it needs ownership of the data.
Data federation works best when the remote platform can handle the requested filters, aggregations, and projections. If the consumer asks for 5 columns from a 100-column table and a narrow date range, source-side filtering can keep transfer small. If a function can’t be pushed down, the engine may need to read far more data. A design that looks efficient on a diagram can become expensive once real SQL patterns appear.
File Or Storage Federation
Storage-level access removes the second query engine from the read path. The consuming engine reads shared object-store files directly, usually with table metadata that identifies snapshots, partitions, and files. This can reduce the coordination cost of remote SQL and can work well for large analytic datasets stored in open formats. It also shifts responsibility toward catalog consistency, storage permissions, file layout, and table-format compatibility.
Zero-copy data integration through shared storage is especially useful when multiple engines need the same analytic data. Each engine can bring its own compute while the underlying files remain in one location. That approach can cut persistent duplication, but it only works safely when the table format and catalog rules prevent readers from seeing partial or inconsistent state.
When You Shouldn’t Ingest Data
The strongest reason to avoid data ingestion is simple: the consumer doesn’t gain enough from owning another copy. If the source is already queryable, governed, current, and sized for the workload, replication may add more operating work than business value. The cases below are where zero-copy data access usually deserves serious consideration.
The Dataset Is Large, But The Useful Slice Is Small
Copying 20 TB so analysts can read a few filtered gigabytes each week is hard to justify. External access lets the query touch only the needed range when the storage layout and engine support pruning. Google Cloud’s external data source guidance specifically calls out direct access to frequently changing external data as a way to avoid reloading it whenever the source changes. The same reasoning applies outside BigQuery: move the result that creates value, not the whole estate by habit.
Data virtualization can also shorten the path for exploratory work. A team evaluating a new source may need 2 weeks of access before it even knows whether the data belongs in a governed model. Building a production pipeline first creates sunk effort and another copy before the use case is proven. Federation gives teams a way to inspect, join, and test the data while the source remains the system of record.
Freshness Matters More Than Locality
Batch data ingestion creates a delay between source change and downstream visibility. Change data capture can narrow that delay, but it still creates a synchronization process that must be watched. If an application needs the source’s current value at query time and can tolerate remote-query latency, leaving the data in place can remove a whole class of stale-copy problems.
This is common for inventory checks, recent transactions, customer status, support context, and operational reference data. Zero-copy data access can be a better fit when the value of a current answer is higher than the value of local storage. The source still needs a clear service level because freshness is useless if the remote system can’t answer during peak demand.
Governance Favors One Authoritative Copy
Every persistent replica creates another place where permissions, retention, deletion requests, masking rules, and audit evidence may need to be managed. Zero-copy data integration can reduce that footprint when the consumer can honor source-side controls or a shared catalog. It doesn’t remove governance work, but it can reduce the number of durable stores that hold the same sensitive rows.
Data virtualization is also useful when ownership is intentionally distributed. A finance domain can retain control of finance tables while a central analytics team gets approved read access. That model fits organizations that want domain ownership without forcing every domain to ship raw copies into a central store.
The Use Case Is Temporary Or Low Frequency
A quarterly investigation, migration audit, one-time reconciliation, or occasional compliance query may not justify a permanent pipeline. Data federation lets teams connect when the question appears, run the work, and remove access when the project ends. The cost profile is then tied more closely to use rather than to a pipeline that runs every hour whether anyone needs the data or not.
This is also where copied pipelines often create hidden maintenance. The first load may be easy, but the pipeline remains after the project ends. Credentials expire, schemas change, and alerts continue to fire. Avoiding the copy can be the cleaner design when the access need is brief.
A Decision Framework For Zero Copy Or Ingestion
A good decision starts with workload facts. Don’t begin with a vendor feature list. Write down how much data the consumer will actually read, how often it will read it, how fresh the answer must be, what happens if the source is unavailable, and whether the consumer needs to modify or retain historical versions. Those details usually make the choice clearer than a generic architecture principle.
Use this table as a first pass. It isn’t a scorecard. A single hard requirement can outweigh the rest.
| Question | Favors Leaving Data In Place | Favors Data Ingestion |
| How much of the source is read? | Small filtered slices | Most rows, repeatedly |
| How fresh must results be? | Current source state | Controlled snapshot is acceptable |
| What query pattern dominates? | Selective reads and exploration | Heavy joins, repeated scans, high concurrency |
| Can the source absorb extra reads? | Yes, with clear limits | No, source workload must be isolated |
| Is write access required? | Read-only is enough | Consumer must update local data |
| Is historical state required? | Source already keeps it | Consumer must preserve its own history |
| Are data copies tightly controlled? | Fewer persistent stores are preferred | Approved local copy is permitted |
| Can remote failure be tolerated? | Yes, or the use case is noncritical | No, local availability is required |
Zero-copy data access is usually strongest when several conditions line up: a capable source, selective queries, high freshness needs, and a preference for fewer copies. Data ingestion becomes more attractive as the workload gets hotter, more repetitive, more transformation-heavy, or more isolated from the source’s operating window.
A hybrid answer is common. Keep cold history and wide fact tables federated, then materialize a small set of hot aggregates or customer features that must answer quickly. Data federation doesn’t need to replace every pipeline to be useful. It only needs to remove the pipelines that exist because copying became a habit.
The Cost Model Behind Leaving Data In Place
Zero copy changes cost rather than making cost disappear. A copied architecture pays for extraction compute, transfer, target storage, transformation, orchestration, and the people who keep those pieces working. A federated design may avoid much of the persistent target storage and recurring load work, but it can add source compute, cross-region transfer, remote-query fees, metadata services, and repeated scans.
Microsoft’s OneLake documentation explains how shortcuts reference data without copying it, which illustrates the storage side of the equation. The reference is cheap compared with another full dataset, but the consuming engine still pays to process what it reads. If 100 dashboards scan the same remote table all day, repeated compute can cost more than materializing a compact reporting table once.
The right comparison is cost per useful workload, not cost per stored terabyte alone. Include source compute, consumer compute, transfer or egress, caching, materialized results, operational support, and failure recovery. Also include the cost of stale data. A cheap pipeline that produces yesterday’s answer can be expensive if the business needs the current state.
Data virtualization often looks attractive for irregular demand because there is no reason to run a full refresh on data that may not be queried. Repeated high-volume demand can reverse that answer. Once the same remote rows are scanned again and again, local materialization may reduce both latency and source cost.
Performance Limits That Change The Answer
Performance is where many zero-copy designs fail after a convincing proof of concept. A demo query usually touches a narrow dataset with one user. Production adds broad joins, dashboard refreshes, concurrent analysts, scheduled jobs, and functions that the remote engine can’t execute. That changes the amount of data crossing the boundary and the number of systems that must be healthy for a query to finish.
Google Cloud’s external table limitations state that external table queries can be slower than queries against standard BigQuery tables and advise loading data when query speed is the priority. That is a useful general rule. Remote access trades some locality and control for freshness and lower duplication. It shouldn’t be treated as a free performance upgrade.
Pushdown Determines How Much Data Moves
The best federated query is selective before the network hop. Filters, column projection, and partial aggregation should run as close to the source as possible. Databricks’ federation performance guidance shows why pushable predicates matter: when a predicate can’t run remotely, more records may need to travel back for local filtering. Teams should test their real SQL, not a simplified benchmark query.
Data federation also inherits source limits. Connection caps, warehouse queues, API throttles, maintenance windows, and network changes can affect consumers. A dashboard that depends on 4 remote systems now has 4 external dependencies. This can be acceptable for exploration and much less acceptable for an executive dashboard with a strict morning deadline.
An in-place design should therefore have a performance budget. Define acceptable p95 latency, concurrency, bytes scanned, source CPU, transfer volume, and failure rate. If the workload exceeds those limits, materialize the hot path rather than forcing every use case through federation.
Governance, Security, And Data Residency
Fewer copies can simplify control, but remote access still creates a data-processing path that must be governed. The consuming platform needs an identity, approved permissions, network access, logging, and a clear policy for what it may cache or materialize. A pointer to sensitive data is still sensitive because it can grant a route to the underlying records.
The NIST Privacy Framework treats data processing as a full life cycle that includes collection, use, sharing, transmission, retention, and disposal. Its privacy risk guidance is useful here because zero-copy data integration changes where processing happens without removing processing itself. Security teams should document which system enforces row or column rules, where query results may persist, how access is revoked, and whether cross-region traffic changes residency obligations.
Data virtualization can help preserve domain ownership when the access layer respects source controls. It can also create confusion if users see a unified catalog and assume all tables have the same policy. Catalog metadata should expose owner, sensitivity, allowed purposes, retention expectations, and contact paths so a logical view doesn’t hide governance differences.
Shortcuts and external references also need their own authorization model. Microsoft’s OneLake shortcut security guidance separates permission to create a shortcut from permission to read the target data. That distinction is important in any zero-copy design. Creating a reference shouldn’t silently expand access beyond what the source owner approved.
Open Table Formats And Storage-Level Access
Open table formats make storage-level federation more practical because they give multiple engines a shared way to understand table state. Files alone aren’t enough. A large lake also needs metadata that tells readers which files belong to the current table version, how schema changes are represented, and which snapshots are valid.
Apache Iceberg’s table specification tracks table state through metadata files, manifests, snapshots, schemas, and partition information. That structure lets independent engines plan reads against the same table without requiring a second warehouse to serve every query. It also supports changes such as schema and partition evolution without forcing a full rewrite for each logical change.
Zero-copy data access over open tables can reduce dependence on a single compute engine. A team may keep governed files in object storage while different engines read them for SQL, AI, or batch work. Google Cloud’s Iceberg managed table documentation describes customer-owned storage that can be read by BigQuery and other engines, which is one example of this shared-storage direction. The main benefit is that compute can change without requiring every change to start with a new full data copy.
Open formats don’t remove coordination needs. Catalog ownership, concurrent writes, snapshot cleanup, file sizing, encryption, and access policy still matter. This form of zero-copy data integration works when those responsibilities are explicit. A pile of Parquet files with no dependable table metadata is a weaker foundation than a governed open table with clear commit rules.
Architecture Patterns That Work In Practice
The most useful production designs mix remote access with selective materialization. A binary rule such as “federate everything” is as risky as “copy everything.” The architecture should follow query temperature, source limits, data sensitivity, and the cost of stale results.
Federate Cold, Materialize Hot
Keep large history tables, raw events, or rarely used attributes in the source. Materialize small aggregates, serving tables, or features that get hit repeatedly. This pattern gives zero-copy data access to the long tail while protecting latency for the hot path. It also keeps the materialized footprint smaller because only proven workloads earn a local copy.
Read Shared Storage With Local Compute
When data already sits in object storage in an open table format, let the consuming engine read those files directly if governance permits. Salesforce Engineering described moving from query federation toward file federation at larger scale to reduce the limits created by coordinating compute on both sides. The broader lesson is that remote SQL and shared storage have different scaling behavior. Choose the access path that matches where the expensive work should run.
Use Federation During Migration
Data federation can bridge old and new platforms while ownership moves gradually. Teams can expose a source through a foreign catalog, validate downstream queries, and materialize only the workloads that need to move. This lowers migration pressure because every source doesn’t need a new pipeline on day 1.
Zero-copy data integration can also support domain boundaries. A customer platform may read approved transaction attributes from a warehouse while the warehouse remains authoritative. A finance model may access product reference data without creating a finance-owned replica of every product field. These patterns work when contracts are explicit and consumers don’t assume remote data has the same performance or retention behavior as local tables.
When Data Ingestion Is Still The Right Choice
Data ingestion remains the better design when the consumer needs predictable local performance, workload isolation, heavy transformation, independent history, or write control. Zero copy should remove unnecessary movement, not block movement that has a clear purpose.
High-concurrency dashboards are a common example. If hundreds of users repeatedly request the same joined dataset, remote queries can hammer the source and pay for the same work many times. A local reporting table or cache can protect the source while giving the BI layer a stable response time. Google Cloud’s BigLake guidance also notes important limitations around external access, which is a reminder that feature parity with local tables can’t be assumed.
Complex transformation is another reason to copy. If a workload requires several passes of cleansing, entity matching, window calculations, enrichment, and historical snapshots, doing all of that remotely on every query wastes compute. Materializing the processed result can make the contract clearer and reduce repeated work. Materialization is especially useful when the transformed output becomes a durable business dataset with its own lifecycle.
The source may also be unsuitable for analytical load. Operational databases are built to serve applications first. Even if a federation connector can reach them, broad scans can compete with transactions. In that case, replication into an analytical store creates workload isolation. The copy has a purpose: protecting the system that runs the business.
A local copy is also appropriate when consumers must work during source outages or when legal retention requires a point-in-time record independent of the source. Zero-copy data access follows the source’s availability and, in many designs, its current state. If the consumer needs a guaranteed historical snapshot, keeping a governed local copy can be the safer choice.
Implementation Checklist For A Zero-Copy Design
A zero-copy project should start with one workload and measurable limits. Avoid opening a large catalog and asking users to discover the failure modes in production. Pick a dataset with a known owner, clear access rules, representative query patterns, and enough traffic to expose real cost and latency behavior.
Before go-live, check the following items:
• Confirm which engine executes filters, joins, and aggregations for the real query set.
• Record bytes scanned, bytes transferred, source compute, consumer compute, and p95 response time.
• Test schema additions, type changes, deleted columns, and renamed fields.
• Verify how row filters, column masking, and source revocation reach the consumer.
• Test source outage behavior and decide whether stale cached results are allowed.
• Define when a federated table should be materialized because traffic or latency crossed a limit.
• Set ownership for catalog metadata, credentials, network routes, and incident response.
• Review whether query results, caches, exports, or downstream models create new persistent copies.
Data virtualization should also be observable as a service. Teams need to know which remote tables are heavily used, which queries fail pushdown, and which sources receive unexpected load. Without that visibility, a low-copy architecture can still create high operational cost.
The final policy should allow movement when the evidence supports it. Zero-copy data integration works best as a default question: “Why do we need another copy?” If the answer is latency, isolation, history, transformation, or availability, create the copy deliberately and govern it as a product rather than as pipeline exhaust.
Conclusion
This model is most valuable when it stops teams from copying data simply because their tools make copying easy. Large, current, well-governed datasets often don’t need to be ingested into every platform that wants to read them. Federation, external tables, secure sharing, and open storage formats can give consumers useful access while the source remains authoritative.
The tradeoff is operational. Remote queries depend on source capacity, network paths, pushdown support, and shared security controls. Repeated hot workloads can cost more to query remotely than to materialize once. Heavy transformation, strict latency, independent history, and source isolation still make data ingestion the right answer in many systems.
A sound architecture uses both patterns. Apply zero-copy data access where a copy adds little value, then materialize only the workloads that prove they need locality. That keeps movement intentional, limits duplicate state, and gives each dataset a reason for every place it lives.
FAQs
What Is Zero-Copy Data Access?
Zero-copy data access lets a system query or use data without first creating a full persistent replica in its own storage. The consumer may keep metadata, permissions, or table references while the authoritative records remain in the source. Query results can still move across the network, and some products may cache selected results.
Is Zero Copy The Same As Data Federation?
No. Data federation is one way to provide zero-copy access. It usually sends queries to a remote source and combines the returned results. Other approaches read shared object-store files directly or expose secure data shares. The common idea is avoiding a required full replica before the consumer can use the data.
Is Data Virtualization The Same As Zero Copy?
They overlap, but the terms aren’t identical. Data virtualization presents remote data through a logical layer and often uses federation underneath. Zero copy is broader and can include direct file access, external tables, shared catalogs, and secure sharing arrangements that don’t depend on a classic virtualization server.
Does Zero Copy Mean No Data Moves Across The Network?
No. Query requests, result rows, file blocks, and intermediate data may still travel across a network. Zero copy usually means the architecture doesn’t require a full persistent duplicate before access. Network transfer remains part of performance, security, and cost planning.
When Should A Team Avoid Data Ingestion?
Avoid ingestion when the source is already fit for querying, users need only a small part of a large dataset, current values matter, access is occasional, or another persistent copy would add governance work without improving the workload. Test real query behavior before making the decision permanent.
When Is Ingestion Better Than Federation?
Ingestion is often better for repeated high-volume scans, high-concurrency dashboards, complex transformations, independent historical snapshots, strict latency targets, or workloads that must continue when the source is unavailable. It can also protect operational systems from analytical load.
Can Zero-Copy Access Reduce Storage Cost?
It can reduce duplicate storage because fewer systems hold full replicas. Total cost still includes source and consumer compute, network transfer, metadata services, caching, and operational support. Repeated remote scans can erase storage savings if the workload is hot and poorly filtered.
How Does Query Pushdown Affect Zero Copy?
Pushdown lets the remote system apply filters, projections, or aggregations before returning data. Good pushdown reduces transfer and often improves latency. Poor pushdown can force the consumer to fetch far more rows than the final answer needs, which can make federation slow and expensive.
Are External Tables Always Slower Than Local Tables?
No, but local tables usually give the engine more control over storage layout, caching, statistics, and execution. External performance depends on source type, file layout, network path, metadata, and query design. Benchmark the real workload rather than assuming either option is faster.
Does Zero Copy Improve Data Freshness?
It can because the consumer reads the current source rather than waiting for a scheduled replica to refresh. Freshness still depends on source commits, metadata refresh behavior, caches, and the platform’s consistency model. Teams should define freshness in measurable terms instead of assuming it is immediate.
Can Zero Copy Help With Data Residency Requirements?
It may help by leaving the main stored dataset in its approved location, but remote access can still involve cross-region queries, temporary processing, logs, or result storage. Residency and privacy requirements should be checked against the full processing path, not just the location of the source files.
Do Open Table Formats Make Zero Copy Easier?
Yes. Formats such as Apache Iceberg give multiple engines a shared definition of table state, snapshots, schema, and file membership. That can make direct storage access safer than reading unmanaged files. Teams still need a dependable catalog, permissions, cleanup policies, and rules for concurrent writes.
Can Zero Copy Replace ETL And ELT Pipelines?
It can remove pipelines whose only job is copying data for read access. It doesn’t replace pipelines that create durable transformed datasets, maintain history, isolate source workloads, or prepare data for strict serving targets. Most mature architectures will keep a mix of federation and materialization.
What Should Teams Monitor In A Zero-Copy Architecture?
Monitor query latency, source compute, consumer compute, bytes scanned, bytes transferred, pushdown rates, source errors, network failures, and cache behavior. Track which remote tables become hot enough to justify materialization. Security logs should also show who queried the source and under which identity.
What Is The Safest Way To Start With Zero Copy?
Start with one read-heavy workload that has a clear owner and moderate traffic. Measure it against an equivalent ingested design, including cost and failure behavior. Expand only after access controls, schema change handling, source limits, and materialization thresholds are understood.
