File system templates store

The default template store implementation persist entities into file system. It is designed to conveniently generate human readable directory structure with templates and other entities definitions. Such structure is very easy to edit with your favorite text editor, deploy to the production and also version with git or other source control. The file system implementation finally supports also cloud based "storage" like AWS S3 or Azure Storage.

Configuration

The default configuration is present in the jsreport.config.json file pre-created during the installation.

"store": { "provider": "fs" }

The fs-store is just another jsreport extension so you can fill other config values to its respective node.

"extensions": {
  "fs-store": {
    "dataDirectory": "data"    
  }
}

The full list of options can be found through help command as always.

jsreport help config

Directory structure

The directory structure generated by fs store is designed to be easy to read. The most of the entities like templates resides in the specific folder which includes config.json file with meta attributes like template recipe or engine. The properties which includes the whole document like template helpers are extracted into dedicated file. This means that you can find for example the template's helpers in the helpers.js.

data \ myfolder

• • invoice
• • • • content.html
• • • • helpers.js
• • • • config.json

Every folder inside the data directory that isn't representing an entity is represented as folder also in the jsreport studio. And every file that isn't inside an entity folder is representing an asset that is also visible in the jsreport studio. Such assets are hover not having extra properties so you can edit only its content. You can use jsreport studio to move folders and entities through directory as well as an external files manager.

Deployment

The deployment of the stored templates to the production server can be done just by copy pasting the data directory. This is also very convenient way to backup the production templates.

Deployment to the running jsreport instance can be also done using cli and import-export without down time and in consistent way. In this case you should integrate to your deployment the following steps.

npm i -g jsreport-cli 

mkdir myjsreportdeployment
cd myjsreportdeployment

jsreport init
git clone https://mygit.com/mytemplates data
jsreport export jsreport-export.zip 

jsreport import jsreport-export.zip --serverUrl https://prod-jsreport.com

Source control

The folder structure created by the fs store is designed to be human readable and easily versioned using source controls like git. The integration is just about including the app/data folder to the source control repository.

Consistency

The fs store uses transnational approach for writing entities to assure that the underlying data keeps consistent even if the process crashes in the middle. Technically it always create the new folder with changed entity and rename it to the final name after everything succeeds.

Running in cluster

The fs store is designed to support running multiple jsreport instances over the same directory when properly configured. It is assured that parallel request doesn't break the data consistency using lock files. The changes made from another process can be automatically monitored and the memory representation reloaded.

The files monitoring and automatic reload can be configure with this config change.

{  
  "extensions": {
    "fs-store": { 
      "syncModifications": {
        "updateStudio": false
      }
    }
  }
}

It is important to understand that if one jsreport instance changes data, every other instance triggers data reload with such configuration. This operation is expensive and it's recommended to limit the data changes, especially the automated changes jsreport does. This can be achieved by extending intervals jsreport uses for data compaction and studio logs flushing.

{  
  "extensions":     
    "fs-store": { "compactionInterval": 60000 }     
    "studio": { "flushLogsInterval": 60000 },
    "scheduling": { "interval": 60000 }  
  }
}

While the files monitoring works in most of the cases, it can have issues in some specific environments. We recommend to spin up multiple instances in your environment and check if the reload works reliably. If the files monitoring isn't reliable you have the following options.

1. Disable templates changes in production using freeze extension and always prepare your templates in dev environment and deploy the final templates to the production. Note this doesn't let you reliably use scheduling or reports because these need to do entities edits in the production.
2. Use service bus to synchronize changes between jsreport instances. Read further bellow how to connect jsreport to AWS SNS or Azure Service Bus for sync. Note you can just use the service bus and don't need to store entities in the S3 or Azure Blob Storage.
3. Use full blown database. See template stores

Editing templates in custom editor

The fs store monitors changes in the underlying files and notifies the studio UI to reload and re-render the active template through sockets. This behavior makes it very easy to edit the templates' code using your editor and immediately see the rendered output in the jsreport studio.

Note this behavior is enabled only in the development environment by default, although you can override this using extensions.fs-store.syncModifications = true config.

You can configure how file monitoring should be done by editing sync.usePolling config. when true it uses polling strategy (checks file stats every x time), when false it uses native os watching mechanism. defaults to true. if you detect some problem with file watching in your installation you can try to set sync.usePolling to false and check if it makes the problem disappear.

"store": {
  "provider": "fs"  
},
"extensions": {
  "fs-store": {
    "dataDirectory": "data",
    "syncModifications": {
      "updateStudio": true
    },
    "sync": {
      "usePolling": true
    }
  }
}

Cloud storage

The fs store by default writes entities to the file system but it also includes persistence abstraction which can be implemented by another extension. Such extension can then provide persistence implementation for another media like AWS S3 or Azure Blob Storage.

jsreport using such extension loads all entities only during the startup and then use in memory representation for queries. This makes typical operations like loading and rendering templates very fast. However it doesn't work in the environment with multiple instances writing to the single storage. Fortunately fs store provides another abstraction used for synchronizing changes between servers. This is technically implemented using cloud native service bus.

Note that the cloud storage is typically slower than disk access. The same what was previously described in cluster section applies here - it is recommended to extend intervals in which jsreport does write.

{  
  "extensions":     
    "fs-store": { "compactionInterval": 60000 }     
    "studio": { "flushLogsInterval": 60000 },
    "scheduling": { "intreval": 60000 }  
  }
}

AWS S3

You need to first create an IAM user with permissions to S3/SNS/SQS and copy the access key with the secret access key. Then create a bucket and copy its name.

To configure jsreport persisting templates into AWS S3 you need to install jsreport-fs-store-aws-s3-persistence extension first.

npm install jsreport-fs-store-aws-s3-persistence

The next step is to configure the fs store persistence using the following options. The jsreport instance should then load and persist templates to specified bucket.

"store": {
  "provider": "fs"
},
"extensions": {
  "fs-store": {
    "persistence": {
      "provider": "aws-s3"
    }
  },
  "fs-store-aws-s3-persistence": {
    "accessKeyId": "...",
    "secretAccessKey": "...",
    "bucket": "....",
    "lock": {
      "queueName": "jsreport-lock.fifo"     
    }
  }
}

To configure jsreport persisting templates in S3 to run in the multi server environment you need to install jsreport-fs-store-aws-sns-sync extension.

npm install jsreport-fs-store-aws-sns-sync

The second step is to configure the fs store synchronization using the following options.

"store": {
  "provider": "fs"
},
"extensions": {
  "fs-store": {
    "persistence": {...},
    "sync": {
      "provider": "aws-sns"
    }
  },
  "fs-store-aws-sns-sync": {
    "accessKeyId": "...",
    "secretAccessKey": "..."   
  }
}

This assures all servers gets a message and reflect it to the memory representation every time there is a change written to the S3. These messages are delivered to the jsreport instances using the native AWS SNS service.

You can find further information in the following repositories.

jsreport-fs-store-aws-s3-persistence
jsreport-fs-store-aws-sns-sync

Azure Storage

You need to create an azure storage account and copy the account name and the access key first.

To configure jsreport persisting templates into Azure Blob Storage you need to install jsreport-fs-store-azure-storage-persistence extension.

npm install jsreport-fs-store-azure-storage-persistence

The second step is to configure the fs store persistence using the following options. The jsreport instance should then load and persist templates to Azure Blob Storage jsreport container.

"store": {
  "provider": "fs"
},
"extensions": {
  "fs-store": {
    "persistence": {
      "provider": "azure-storage"
    }
  },
  "fs-store-azure-storage-persistence": {
    "accountName": "...",
    "accountKey": "..."  
  }
}

To configure jsreport persisting templates into Azure Blob Storage and run in the multi server environment you need to first install jsreport-fs-store-azure-sb-sync extension.

npm install jsreport-fs-store-azure-sb-sync

Then create an azure service bus and copy the connection string from the shared access policies. The next step is to configure the fs store synchronization using the following options.

"store": {
  "provider": "fs"
},
"extensions": {
  "fs-store": {
    "sync": {
      "provider": "azure-sb"
    }
  },
  "fs-store-azure-sb-sync": {
    "connectionString": "..."    
  }
}

This assures all servers gets a message and reflect it to the memory representation every time there is a change written by one of the instance to the Azure Blob Storage. These messages are delivered to the jsreport instances using the native Azure Service Bus.

You can find further information in the following repositories.

jsreport-fs-store-azure-storage-persistence jsreport-fs-store-azure-sb-sync