Click or drag to resize
Import and Export

[This is preliminary documentation and is subject to change.]

Import and export is a very broad term. It may be implemented in a number of ways for TOPICA based applications

Explore the possibilities described below, select the method approriate for your use, and read further descriptions on the selected method(s) by following the links.

Built-in import/export functionality

A few import/export functions are provided by the framework. These are naturally limited to the types of data, that the framework manages. Currently the following data may be imported/exported:

These functions are available to end users having approriate permissions.

All above functions use XML files for data exchange. The use of the XML file format makes it possible to move data from one environment to another, even if there is no network connection between the environments. The user exports data from environment 1, saves the XML file locally, and imports the XML file into environment 2. This method may thus be used to move a complete organization between for example a develoment environment and a production environment.

There is a SQL Query function built into the TOPICA framework (available for system administrators only). By entering SQL queries in this function, it is possible to extract any data from the database - whether is is stored in the static or the dynamic data structure. The data is extracted as separated text files (comma-, semicolon- or tab-separated text files).

Import/export using webservices

Most types of data managed by the framework (e.g. data in the static data model) may be read and written using ""static" webservices built into the framework:

  • Organizational units are managed by OrgUnit service.

  • Employees (users) are managed by Employee service. '

  • Patients are managed by Patient Service.

    Patient data (records) are stored in the dynamic data model. These data may be read/written by a set of "generic" websevices (working on key/value pairs). defined in Patient Service.

    However, it also is possible to have webservices for a specific dynamic data model generated automatically. I.e. each form type (= table in the data dictionary) generates webservice methods to create, read, update and delete operations. To automatically create a dynamic webservice see Integrations

Using webservices requires some programming. I.e. this is a task for an application developer

A typical scenario for using TOPICA webservices for import/export, is when developing interfaces to external systems. Obviously only in the case, where an external system is the "master", and needs to update the TOPICA programmatically.

For example, when using TOPICA to build modules in context manager applications, it is often necessary to develop some back-end integration in order to synchronize organization and users in the TOPICA database to match the corresponding data in the context manager application's database.

Refer to Web Services for further information regarding use of TOPICA webservices.

Import/export on the database level

It is possible to use standard tools like SQL queries/inserts/updates, SSIS, etc. to perform import/export on the database level. Obviously this requires solid knowledge of the data model.

Security note Security Note

Security, rules, etc. are NOT checked. This option is for system administrators only.

A common scenario for using this technique, is to implement "bulk" import from files (comma-separated texts files, XML files, Excel spreadsheets, etc.) generated by external systems. TOPICA contains "building blocks" making it easier to implement import/export:

  • Functionality to upload files from client to web server (optionally further upload to a shared FTP site).

  • Possibility to execute SSIS-packages on the web server. Such packages read file(s) uploaded to the web server, and use functionality in SSIS to extract data from the import file(s) and update/insert into the database.

  • Possibility to execute SSIS-packages stored in the database. Such packages copy the uploaded files from the shared FTP site to the database server - and use functionality in SSIS to extract data from the import file(s) and update/insert into the database.

Export using reports

TOPICA internal format reports may export in comma-, semicolon- or tab-separated text file format.

Reporting Services reports may be integrated into a TOPICA based application. All Reporting Services export formats may be used. Reporting Services 2008 R2 support these formats:

  • Comma-separated format

  • XML

  • TIFF

  • Excel

  • Word

  • PDF

Refer to Guide to configuring reports and Configuration reference - Reports for further information on how to develop reports.

Trade-offs

As can be seen from the above list, there are several ways to implement import/export functionality. More often than not a given functionality may be implemented in more than one way. It is crucial to select the right tool for the job. Here are some trade-offs to be considered:

SSIS-packages should be the first choice for "bulk" operations. It is the most flexible way to do import/export. SSIS is Microsoft's standard ETL (Extract/Transform/Load) tool. SSIS comes bundled with the SQL SErver installation. The SSIS tools in Visual Studio (workflow and dataflow editors etc.) are very flexible, containing standard tools for data flow, error handling, fuzzy logic, logging, etc. Most import/export tasks may be implemented without coding (of course SQL is needed to get data in and out of databases in most cases).

Developing plug-ins for "bulk" import/export is more or less a deprecated feature. It requires developing .NET assemblies implementing interfaces defined in TOPICA. In order to do the required jobs, most plugins use the TOPICA API to read/write data. This means, that plugins get close ties to the TOPICA framework. New versions of the framework may break existing plugins (rewriting / recompiling may be needed). The DLL files implementing the plugins also add to deployment complexity.

Built-in webservices and custom webservices may be used to implement import/export, when controlled from an external system (which may be a program developed for the specific purpose). However, this should be limited to relatively small amounts of data - e.g. in a messaging system, you could develop a custom webservice to handle receiving messages (one message at a time). For "bulk" import/export, the webservice mechanism is far less efficient than for example SSIS-package.