Dynamic table names

In our source a new table gets created every month with standard such as TABLE_MMYY. Please let me know if there is any way we could pass this dynamic table name to Stichdata as part of extraction?.

We can not use a view definition on top of the tables because there are 100’s of such monthly tables and a view definition would blow out.

Hey there,

As a heads up for future reference in case you haven’t already reached out, the best place to get a response directly from Stitch’s support team would be via our in-application chat or via email at stitchsupport@talend.com, which sends questions or concerns directly to the team.

With regard to handling a dynamic table name as described in your use-case, the answer will ultimately depend on the data source you’re working with, though our SaaS integrations usually have hard-coded schemas and there’s only a few exceptions to the behavior I’ve outlined below that are specific to data lake or file storage services.

For database sources, as well as our Salesforce integration, the source integration within Stitch would identify each of these as new tables because information about the configuration and bookmarks for a source integration are dependent on the search path of the source table.

In terms of how that affects table / field selection, each of these tables would need to be selected for replication, have fields selected, and have a replication method defined whenever they’re created in the source instance, and Stitch doesn’t currently support an approach to automatically track new tables for replication.

In case your larger concern was on the destination side, this functionality also means that as the data from each new table is being loaded, each would be loaded into a distinct table in your destination, so the resource consumption on the destination side may grow.

The only exceptions to these rules would be our S3-CSV source integration and our SFTP source integration, which allow you to define the search pattern that the integration should use to identify which files correspond to which table.

If you are working with a database source or Salesforce, there’s not a way to provide a dynamic table name as you’ve described, but one option would be to use our Connect API that’s accessible as part of an Enterprise plan, where you could build a script to interact with endpoints surrounding table and field selection for your data source programatically. An example of a workflow using it could be adding some steps to the end of your existing process for generating these tables in the source instance that makes posts against the API to have the integration run a connection check to detect the new tables, select / configure the tables and fields that were added, and even run a new extraction job after the tables have been selected. This wouldn’t change the behavior on the destination side, but would eliminate the process of manually selecting each of the tables after they’ve been added.

Depending on how you’re hoping to have this situation handled in your destination, our post-load-webhooks (also accessible as part of an Enterprise plan) could be used to send events surrounding successful loading commands run against your destination, which could trigger a script you manage on the destination side. Some examples of processes you trigger using the webhook might include merging older tables into the most recent table, dropping the previous versions of the table, or directing any data models or reporting to the most recent table. Beyond this, there’s a number of other Talend products available that could be used in conjunction with Stitch to define transformations or modeling downstream.

1 Like