Phase 3: Artifact migration

Running a bulk import

Whichever platform you are leaving, a back-filling migration, that is, moving existing packages out of the old platform in bulk, ends the same way: exported packages pushed into Cloudsmith with the Cloudsmith CLI, or container images copied registry to registry. This page is the runbook for that work. It covers the order to do things in, the decisions to make before the first push, and how to verify the result. Importing packages with the CLI covers what each package type needs.

The runbook

Work through your source repositories one at a time, smallest first. Each pass is the same:

1. Pick the next source repository and decide which Cloudsmith repository it maps to. See Migrating from JFrog Artifactory or Migrating from Sonatype Nexus for what does and does not need moving.
2. Export it to a folder on a machine with enough disk for that one repository. Run the export and import from a server or VM close to the source platform, not a laptop: a large repository takes hours to days and the machine must stay up throughout. Docker images need no export; they are copied directly.
3. Import it with the CLI scripts, or copy it if it is a container registry, tagging every package with the source repository name.
4. Verify every recorded identifier and the totals as described below, fix or retry anything that failed, then delete the export folder.
5. Repeat for the next repository. When all are done, move on to Phase 4: Validation and decommissioning.

Start with a pilot: one small repository taken through all five steps. It shows how long a gigabyte takes on your network, whether the machine running the import has the disk and rate limits it needs, and what the verification numbers look like when everything works. Only then scale up.

Tip

Total size matters less than file count. Twenty terabytes of container images copies in the background with no local storage. Twenty terabytes of small npm tarballs is millions of uploads, and the rate limit and sync queue, not bandwidth, set the pace. Estimate from the pilot before committing to a cutover date.

Use a dedicated service account

Run the import as a service account created for the migration, rather than as a personal user. Its activity is then easy to separate in audit logs and package listings, and the API key can be revoked when the migration is complete. The account needs write privileges on every repository it uploads to.

Export the key before running any script:

export CLOUDSMITH_API_KEY=your-service-account-api-key

Tag packages with their origin

Pass --tags on every push so migrated packages can be found later, for example to compare counts against the source or to clean up a failed run. A tag for the migration and one for the source repository is usually enough. The import scripts in Importing packages with the CLI read the tags from a TAGS variable:

TAGS=migration,npm-local ./push-folder.sh npm my-org/my-repo ./export/npm-local tgz

Tags are searchable with tag:migration in the web app, the CLI, and the API. See Package tags.

Do not wait for synchronization

After upload, Cloudsmith processes each package before it is available to install. By default the CLI waits for that to finish. For a bulk import pass --no-wait-for-sync (-W) so the CLI returns as soon as the upload completes and the packages queue in Cloudsmith rather than on the machine running the migration.

The CLI still prints the new package's identifiers, Created: OWNER/REPO/SLUG (SLUG_PERM), or the same as JSON with -F json. Record the slug_perm of every push. It is the one thing that ties a file on disk to a package in Cloudsmith, and it is how you verify the import without guessing. The import scripts write it to pushed.tsv for you, one line per upload: the file, its slug_perm, and the repository it went to.

Note

With --no-wait-for-sync, packages are not installable until they leave the queue, and on a large import the queue can take a while to drain. Verify at the end, from the recorded identifiers, rather than after each push.

Know your rate limits

API requests are rate limited per account, and the limit depends on your plan. Check the limit that applies to the account you are using:

cloudsmith check rates

The CLI respects the limit automatically and backs off on 429 responses, so a long-running script slows down rather than failing. Running a few pushes in parallel helps with many small files; more than about four workers gains little. See Pushing in parallel. Each Workspace also has a throttle on how quickly it can accept uploads. If you plan to import a large number of packages, tell the Cloudsmith team when the migration will run and roughly how many packages it involves, so the limits can be raised for that period and the platform can be monitored. See Rate limits and scaling.

Verify the import

Once the queue has drained, check every package you pushed by its recorded identifier. cloudsmith status answers for one package; the loop in Importing packages with the CLI runs it over pushed.tsv and prints anything that is not Completed.

cloudsmith status my-org/my-repo/SLUG_PERM

Then compare totals with what you exported. Count the files in the source repository before exporting (the Artifactory and Nexus guides show how), then count what carries its tag in Cloudsmith. The same package search syntax works in the CLI, the web app, and the packages API:

# Everything tagged from that repository, and anything that failed
cloudsmith list packages my-org/my-repo -q 'tag:npm-local'
cloudsmith list packages my-org/my-repo -q 'tag:npm-local status:failed'

The numbers do not have to match exactly. Formats such as Maven turn several files into one package, and duplicates in the source collapse into one package in Cloudsmith. They do have to be explainable.

When scripting against the API, read the status field of each package. Only 4 (Completed) means the package is ready to install.

Failed packages are removed automatically after 24 hours, so check for failures within a day of the import. The most common failure is a duplicate: a package with the same name and version already exists and the repository does not allow overwrites. That makes re-running an import safe. Existing packages are left alone and only the missing ones land. Pass --republish if you want the new upload to win instead. See Package search syntax for the full set of filters.

Commands you will use

Everything below is done with a handful of CLI commands. The scripts in Importing packages with the CLI are loops around the second row.

Format differences

Most formats package everything into one file, so one cloudsmith push per file is all you need. A few need more. Importing packages with the CLI has the script and a section for each case:

Next steps

Export your packages with the guide for your platform, JFrog Artifactory or Sonatype Nexus, then import them with Importing packages with the CLI or Importing Docker images.

Updated 1 minute ago