Since each replica in swift functions independently, and clients generally require only a simple majority of nodes responding to consider an operation successful, transient failures like network partitions can quickly cause replicas to diverge. These differences are eventually reconciled by asynchronous, peer-to-peer replicator processes. The replicator processes traverse their local filesystems, concurrently performing operations in a manner that balances load across physical disks.
Replication uses a push model, with records and files generally only being copied from local to remote replicas. This is important because data on the node may not belong there (as in the case of handoffs and ring changes), and a replicator can’t know what data exists elsewhere in the cluster that it should pull in. It’s the duty of any node that contains data to ensure that data gets to where it belongs. Replica placement is handled by the ring.
Every deleted record or file in the system is marked by a tombstone, so that deletions can be replicated alongside creations. These tombstones are cleaned up by the replication process after a period of time referred to as the consistency window, which is related to replication duration and how long transient failures can remove a node from the cluster. Tombstone cleanup must be tied to replication to reach replica convergence.
If a replicator detects that a remote drive is has failed, it will use the ring’s “get_more_nodes” interface to choose an alternate node to synchronize with. The replicator can generally maintain desired levels of replication in the face of hardware failures, though some replicas may not be in an immediately usable location.
Replication is an area of active development, and likely rife with potential improvements to speed and correctness.
复制是一个活跃的开发领域，在速度和正确性上具有提升的潜力。
There are two major classes of replicator - the db replicator, which replicates accounts and containers, and the object replicator, which replicates object data.
The first step performed by db replication is a low-cost hash comparison to find out whether or not two replicas already match. Under normal operation, this check is able to verify that most databases in the system are already synchronized very quickly. If the hashes differ, the replicator brings the databases in sync by sharing records added since the last sync point.
This sync point is a high water mark noting the last record at which two databases were known to be in sync, and is stored in each database as a tuple of the remote database id and record id. Database ids are unique amongst all replicas of the database, and record ids are monotonically increasing integers. After all new records have been pushed to the remote database, the entire sync table of the local database is pushed, so the remote database knows it’s now in sync with everyone the local database has previously synchronized with.
In practice, DB replication can process hundreds of databases per concurrency setting per second (up to the number of available CPUs or disks) and is bound by the number of DB transactions that must be performed.
The initial implementation of object replication simply performed an rsync to push data from a local partition to all remote servers it was expected to exist on. While this performed adequately at small scale, replication times skyrocketed once directory structures could no longer be held in RAM. We now use a modification of this scheme in which a hash of the contents for each suffix directory is saved to a per-partition hashes file. The hash for a suffix directory is invalidated when the contents of that suffix directory are modified.
The object replication process reads in these hash files, calculating any invalidated hashes. It then transmits the hashes to each remote server that should hold the partition, and only suffix directories with differing hashes on the remote server are rsynced. After pushing files to the remote server, the replication process notifies it to recalculate hashes for the rsynced suffix directories.
Performance of object replication is generally bound by the number of uncached directories it has to traverse, usually as a result of invalidated suffix directory hashes. Using write volume and partition counts from our running systems, it was designed so that around 2% of the hash space on a normal node will be invalidated per day, which has experimentally given us acceptable replication speeds.
Rate limiting in swift is implemented as a pluggable middleware. Rate limiting is performed on requests that result in database writes to the account and container sqlite dbs. It uses memcached and is dependent on the proxy servers having highly synchronized time. The rate limits are limited by the accuracy of the proxy server clocks.
All configuration is optional. If no account or container limits are provided there will be no rate limiting. Configuration available:
所有的配置是可选的。如果没有给出账号或容器的限制，那么就没有速率限制。可用配置参数如下：
Option
Default
Description
clock_accuracy
1000
Represents how accurate the proxy servers’ system clocks are with each other. 1000 means that all the proxies’ clock are accurate to each other within 1 millisecond. No ratelimit should be higher than the clock accuracy.
To allow visibility into rate limiting set this value > 0 and all sleeps greater than the number will be logged.
在速率限制中考虑可见性，设置这一值大于0并且所有的休眠时间大于这个数值得将被记录。
rate_buffer_seconds
5
Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy.
Comma separated lists of account names that will not be rate limited.
由逗号分隔的不会被速度限制的账号名字列表。
account_blacklist
‘’
Comma separated lists of account names that will not be allowed. Returns a 497 response.
由逗号分隔的不被允许的账号名称列表。
container_ratelimit_size
‘’
When set with container_limit_x = r: for containers of size x, limit requests per second to r. Will limit PUT, DELETE, and POST requests to /a/c/o.
当设置为container_limit_x = r :对于大小为x的容器，限制的请求数为r次每秒。使用/a/c/o来限制PUT,DELETE和POST请求。
The container rate limits are linearly interpolated from the values given. A sample container rate limiting could be:
容器的速率限制从给定值线性地插入。一个容器速率限制的样例如下：
container_ratelimit_100 = 100
container_ratelimit_200 = 50
container_ratelimit_500 = 20
This would result in
复制代码
这将会产生
Container Size
Rate Limit
0-99
No limiting
100
100
150
75
500
20
1000
20
7. Large Object Support 大对象支持
7.1 Overview 概述
Swift has a limit on the size of a single uploaded object; by default this is 5GB. However, the download size of a single object is virtually unlimited with the concept of segmentation. Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as a single object. This also offers much greater upload speed with the possibility of parallel uploads of the segments.
7.2 Using swift for Segmented Objects 使用swift来分割对象
The quickest way to try out this feature is use the included swift Swift Tool. You can use the -S option to specify the segment size to use when splitting a large file. For example:
swift upload test_container -S 1073741824 large_file
This would split the large_file into 1G segments and begin uploading those segments in parallel. Once all the segments have been uploaded, swift will then create the manifest file so the segments can be downloaded as one.
So now, the following swift command would download the entire large object:
所以现在，使用以下swift命令可以下载整个大对象：
swift download test_container large_file
swift uses a strict convention for its segmented object support. In the above example it will upload all the segments into a second container named test_container_segments. These segments will have names like large_file/1290206778.25/21474836480/00000000, large_file/1290206778.25/21474836480/00000001, etc.
The main benefit for using a separate container is that the main container listings will not be polluted with all the segment names. The reason for using the segment name format of <name>/<timestamp>/<size>/<segment> is so that an upload of a new file with the same name won’t overwrite the contents of the first until the last moment when the manifest file is updated.
swift will manage these segment files for you, deleting old segments on deletes and overwrites, etc. You can override this behavior with the --leave-segments option if desired; this is useful if you want to have multiple versions of the same large object available.
You can also work with the segments and manifests directly with HTTP requests instead of having swift do that for you. You can just upload the segments like you would any other object and the manifest is just a zero-byte file with an extra X-Object-Manifest header.
All the object segments need to be in the same container, have a common object name prefix, and their names sort in the order they should be concatenated. They don’t have to be in the same container as the manifest file will be, which is useful to keep container listings clean as explained above with swift.
The manifest file is simply a zero-byte file with the extra X-Object-Manifest: <container>/<prefix> header, where <container> is the container the object segments are in and <prefix> is the common prefix for all the segments.
It is best to upload all the segments first and then create or update the manifest. In this way, the full object won’t be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.
# And now we can download the segments as a single object
curl -H 'X-Auth-Token: <token>' \
http://<storage_url>/container/myobject
复制代码
7.4 Additional Notes 其他注意事项
With a GET or HEAD of a manifest file, the X-Object-Manifest: <container>/<prefix> header will be returned with the concatenated object so you can tell where it’s getting its segments from.
The response’s Content-Length for a GET or HEAD on the manifest file will be the sum of all the segments in the <container>/<prefix>listing, dynamically. So, uploading additional segments after the manifest is created will cause the concatenated object to be that much larger; there’s no need to recreate the manifest file.
The response’s Content-Type for a GET or HEAD on the manifest will be the same as the Content-Type set during the PUT request that created the manifest. You can easily change the Content-Type by reissuing the PUT.
The response’s ETag for a GET or HEAD on the manifest file will be the MD5 sum of the concatenated string of ETags for each of the segments in the <container>/<prefix> listing, dynamically. Usually in Swift the ETag is the MD5 sum of the contents of the object, and that holds true for each segment independently. But, it’s not feasible to generate such an ETag for the manifest itself, so this method was chosen to at least offer change detection.
If you are using the container sync feature you will need to ensure both your manifest file and your segment files are synced if they happen to be in different containers.
如果你选择了容器同步的特性，你将需要来确保你的描述文件和你的分卷文件被同步若它们在不同的容器中。
7.5 History 发展史
Large object support has gone through various iterations before settling on this implementation.
大对象的支持在设为现在这种实现方式前已经经历了各种反复修改。
The primary factor driving the limitation of object size in swift is maintaining balance among the partitions of the ring. To maintain an even dispersion of disk usage throughout the cluster the obvious storage pattern was to simply split larger objects into smaller segments, which could then be glued together during a read.
Before the introduction of large object support some applications were already splitting their uploads into segments and re-assembling them on the client side after retrieving the individual pieces. This design allowed the client to support backup and archiving of large data sets, but was also frequently employed to improve performance or reduce errors due to network interruption. The major disadvantage of this method is that knowledge of the original partitioning scheme is required to properly reassemble the object, which is not practical for some use cases, such as CDN origination.
In order to eliminate any barrier to entry for clients wanting to store objects larger than 5GB, initially we also prototyped fully transparent support for large object uploads. A fully transparent implementation would support a larger max size by automatically splitting objects into segments during upload within the proxy without any changes to the client API. All segments were completely hidden from the client API.
This solution introduced a number of challenging failure conditions into the cluster, wouldn’t provide the client with any option to do parallel uploads, and had no basis for a resume feature. The transparent implementation was deemed just too complex for the benefit.
The current “user manifest” design was chosen in order to provide a transparent download of large objects to the client and still provide the uploading client a clean API to support segmented uploads.
Alternative “explicit” user manifest options were discussed which would have required a pre-defined format for listing the segments to “finalize” the segmented upload. While this may offer some potential advantages, it was decided that pushing an added burden onto the client which could potentially limit adoption should be avoided in favor of a simpler “API” (essentially just the format of the ‘X-Object-Manifest’ header).
During development it was noted that this “implicit” user manifest approach which is based on the path prefix can be potentially affected by the eventual consistency window of the container listings, which could theoretically cause a GET on the manifest object to return an invalid whole object for that short term. In reality you’re unlikely to encounter this scenario unless you’re running very high concurrency uploads against a small testing environment which isn’t running the object-updaters or container-replicators.
Like all of swift, Large Object Support is living feature which will continue to improve and may change over time.
像所有的swift版本，大对象支持是一个活跃的特性，将会继续改进并且不断地改变。
8. Container to Container Synchronization 容器同步
8.1 Overview 概述
Swift has a feature where all the contents of a container can be mirrored to another container through background synchronization. Swift cluster operators configure their cluster to allow/accept sync requests to/from other clusters, and the user specifies where to sync their container to along with a secret synchronization key.
Container sync will sync object POSTs only if the proxy server is set to use “object_post_as_copy = true” which is the default. So-called fast object posts, “object_post_as_copy = false” do not update the container listings and therefore can’t be detected for synchronization.
If you are using the large objects feature you will need to ensure both your manifest file and your segment files are synced if they happen to be in different containers.
如果你使用大对象特性你将需要确保你的描述文件和你的分卷文件被同步了，如果它们在不同的容器内。
8.2 Configuring a Cluster’s Allowable Sync Hosts 配置一个集群容许的同步主机
The Swift cluster operator must allow synchronization with a set of hosts before the user can enable container synchronization. First, the backend container server needs to be given this list of hosts in the container-server.conf file:
# This is a comma separated list of hosts allowed in the
# X-Container-Sync-To field for containers.
# allowed_sync_hosts = 127.0.0.1
allowed_sync_hosts = host1,host2,etc.
...
[container-sync]
# You can override the default log routing for this app here (don't
# use set!):
# log_name = container-sync
# log_facility = LOG_LOCAL0
# log_level = INFO
# Will sync, at most, each container once per interval
# interval = 300
# Maximum amount of time to spend syncing each container
# container_time = 60
复制代码
Tracking sync progress, problems, and just general activity can only be achieved with log processing for this first release of container synchronization. In that light, you may wish to set the above log_ options to direct the container-sync logs to a different file for easier monitoring. Additionally, it should be noted there is no way for an end user to detect sync progress or problems other than HEADing both containers and comparing the overall information.
The authentication system also needs to be configured to allow synchronization requests. Here is an example with TempAuth:
认真系统也需要进行配置来允许同步请求。这里有一个关于TempAuth的例子：
[filter:tempauth]
# This is a comma separated list of hosts allowed to send
# X-Container-Sync-Key requests.
# allowed_sync_hosts = 127.0.0.1
allowed_sync_hosts = host1,host2,etc.
复制代码
The default of 127.0.0.1 is just so no configuration is required for SAIO setups – for testing.
默认值127.0.0.1只是因为SAIO设置不需要配置-用于测试。
8.3 Using the swift tool to set up synchronized containers 使用swift工具来设置同步容器
Note 注意
You must be the account admin on the account to set synchronization targets and keys.
你必须使用该帐号的帐号管理权限来设置同步目标和键。
You simply tell each container where to sync to and give it a secret synchronization key. First, let’s get the account details for our two cluster accounts:
The -t indicates the URL to sync to, which is the StorageURL from cluster2 we retrieved above plus the container name. The -k specifies the secret key the two containers will share for synchronization. Now, we’ll do something similar for the second cluster’s container:
That’s it. Now we can upload a bunch of stuff to the first container and watch as it gets synchronized over to the second:
就是如此。现在我们可以上载一堆东西到第一个容器并且观察它与第二个容器进行同步：
$ swift -A http://cluster1/auth/v1.0 -U test:tester -K testing \
upload container1 .
photo002.png
photo004.png
photo001.png
photo003.png
复制代码
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 \
list container2
复制代码
[Nothing there yet, so we wait a bit...]
[If you're an operator running SAIO and just testing, you may need to
$ swift -A http://cluster2/auth/v1.0 -U test2:tester2 -K testing2 \
list container2
photo001.png
photo002.png
photo003.png
photo004.png
复制代码
You can also set up a chain of synced containers if you want more than two. You’d point 1 -> 2, then 2 -> 3, and finally 3 -> 1 for three containers. They’d all need to share the same secret synchronization key.
8.4 Using curl (or other tools) instead 使用curl(或其他工具代替)
So what’s swift doing behind the scenes? Nothing overly complicated. It translates the -t <value> option into an X-Container-Sync-To:<value> header and the -k <value> option into an X-Container-Sync-Key: <value> header.
8.5 What’s going on behind the scenes, in the cluster? 在集群中，后台正在运行着什么？
The swift-container-sync does the job of sending updates to the remote container.
swift-container-sync执行发送update到远程容器的工作。
This is done by scanning the local devices for container databases and checking for x-container-sync-to and x-container-sync-key metadata values. If they exist, newer rows since the last sync will trigger PUTs or DELETEs to the other container.
Container sync will sync object POSTs only if the proxy server is set to use “object_post_as_copy = true” which is the default. So-called fast object posts, “object_post_as_copy = false” do not update the container listings and therefore can’t be detected for synchronization.
The actual syncing is slightly more complicated to make use of the three (or number-of-replicas) main nodes for a container without each trying to do the exact same work but also without missing work if one node happens to be down.
Two sync points are kept per container database. All rows between the two sync points trigger updates. Any rows newer than both sync points cause updates depending on the node’s position for the container (primary nodes do one third, etc. depending on the replica count of course). After a sync run, the first sync point is set to the newest ROWID known and the second sync point is set to newest ROWID for which all updates have been sent.
An example may help. Assume replica count is 3 and perfectly matching ROWIDs starting at 1.
一个例子有助于理解。假设replica数量是3并且完全匹配在1开始的ROWID。
First sync run, database has 6 rows:
SyncPoint1 starts as -1.
SyncPoint2 starts as -1.
No rows between points, so no “all updates” rows.
Six rows newer than SyncPoint1, so a third of the rows are sent by node 1, another third by node 2, remaining third by node 3.
SyncPoint1 is set as 6 (the newest ROWID known).
SyncPoint2 is left as -1 since no “all updates” rows were synced.
Next sync run, database has 12 rows:
SyncPoint1 starts as 6.
SyncPoint2 starts as -1.
The rows between -1 and 6 all trigger updates (most of which should short-circuit on the remote end as having already been done).
Six more rows newer than SyncPoint1, so a third of the rows are sent by node 1, another third by node 2, remaining third by node 3.
SyncPoint1 is set as 12 (the newest ROWID known).
SyncPoint2 is set as 6 (the newest “all updates” ROWID).
In this way, under normal circumstances each node sends its share of updates each run and just sends a batch of older updates to ensure nothing was missed.
/2 
京ICP备2020039040号
简书网举报电话：021-34700000
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡
