Create an External SFTP Data Source

Configure an External SFTP  Data Source to extract data from an external External SFTP  Location.

What is an External SFTP  Data Source?

We use External SFTP  Data Sources to connect to any External SFTP  server hosted outside One Model that stores your data. This is separate to Internal SFTP  Users for a File Data Source, which allow users to upload data to a One Model hosted SFTP  Server.

Create an External SFTP  Data Source

To Create an External SFTP  Data Source, go to the Data Sources page and select External SFTP  from the Add Data Sources menu:

From there you'll be able to configure these options:

• Name: The name of the External SFTP  Data Source
• Data Loads Should Process Data: enable if you want Model Processing to run once Data has loaded
• Host: The Host URL of the External SFTP  Server being connected to
• Port: The Port number of the External SFTP  (usually 22)

Choose Password as an Authentication method if you wish to use a user name and password

• Username: The username configured on the External SFTP  Server
• Password: The password configured for the user on the External SFTP  Server

Choose private key as an authentication method if you wish to use a private key

• Private key: The private key of the user account on your sftp server. It can be encrypted with a passphrase, or unencrypted.
• Private key passphrase : The passphrase for an encrypted private key. This passphrase is optional and only required when the private key is encrypted. If the private key is unencrypted then there is no need for a passphrase. 

Add Files to your External SFTP  Data Source

Once you've created your External SFTP  Data Source, you can add some files. There are four kinds of files that can be added:

• Delimited Input: This is for flat files
• JSON Input: This is for files written in JSON, which may contain multiple objects/tables
• XML Input: This is for files written in XML, which may also contain multiple objects/tables
• Compressed File input: This is for cases where a single Compressed archive (like a Zip File) contains more than one file of the above three types

Configuration Options available for all Input types:

In all Input types, the following configuration options are possible:

• External SFTP FilePath: This is the path for the folder on the External SFTP  Server that the desired files are located. Use “/<myFolderName>/” for files contained in folders, or “/” for files not contained in folders.
• External SFTP FileNamePattern: This is the expected name for the pattern. A % can be used as a wildcard to denote part of the name that changes (such as a timestamp). This is only necessary for the ExactFileNameMatch Match Method
• External SFTP MatchMethod: This is the method used to identify files that need to be pulled when the External SFTP  Data Source is run.
• External SFTP NewSinceUtc: The recommended format is yyyy-MM-ddTHH:mm:ss.fffffffZ

Add Compressed Folder Input

Note: This is only necessary for cases where a single compressed file contains multiple files of another type. While this is available for the External SFTP  Data Source, it is currently not available for the File Data Source

To add a Compressed Folder Input, simply configure the options specified above to find the file name. The only specific configuration option for the Compressed Folder input is:

• addCompressedFolderNameToFiles: True/False. If set to True, this adds the Compressed Folder name to all files contained within the file (before the extension)
• The "extractFromCompressedFolder" match method is used when a compressed file contains multiple files of another type, such as delimited JSON or XML. It allows the system to extract those individual files from the compressed folder and process them based on their specific file input configurations.

Add Delimited File Input

Further details for how to use Delimited File Inputs can be found here.

If you have questions about External SFTP , be sure to check out our External SFTP  FAQs here.