Nearly two years ago we launched LDC Via, a new platform that allows you to migrate your legacy Notes and Domino data, and host it on our servers while retaining a lot of the best features of Lotus Notes. Over that time we’ve talked to a lot of people and, it’s fair to say, what we offer is a broad palette of tools: a migration utility, a database platform and a full featured API. But it can be a little overwhelming to get started.
Initially we created several templates that run on the main LDC Via servers. But they were really technical demonstrations at best. So, over the last few months, we’ve created a new service: KEEP.WORKS.
KEEP.WORKS offers several things. Key among them is a simple migration path. You simply need to upload your legacy Notes database to our servers and we will convert it for you. This can be done in as little as ten minutes, depending on the size of your database. If your database matches one of our standard templates (Discussion, Document Library, Teamroom, Journal, Personal Address Book or Mail) then you can immediately log in and carry on working with your data.
To be honest, if that were all that KEEP.WORKS offered you might feel a little underwhelmed. Automated database migration is nice, but we want more! Well, how about a configuration driven way of continuing to work with your data? We’ve got it.
When you import a non-standard database, you can configure how to view, create and edit the data. You can even enable document-level security and set up additional users to access it. All without writing a single line of code.
And the nice thing about all this? Well it’s developed using LDC Via as the underlying platform, so if you outgrow the KEEP.WORKS platform your developers can continue to work with the data and create your own custom applications using whatever languages and coding tools you (or they!) choose. Or, if you prefer, we can create more complex custom applications for you.
For the first month of its life, KEEP.WORKS is going to run in an invitation-only beta status, with a strict limit of 12 places. If you’d like to be one of the first people to put KEEP.WORKS through its paces, visit the home page here: https://keep.works/ - alternatively, you’ll be able to sign up for a full account in just a couple of months time.
Today we are announcing a brand new product: KEEP.WORKS.
KEEP.WORKS lowers the barrier for migrating IBM Notes and Domino data to a modern cloud platform. For a very small monthly fee, an individual or a small team can upload an NSF file, and have it converted automatically and available on desktop and mobile devices for them and any other users they care to invite. That’s really just the tip of the iceberg, and more details will follow.
But for now… we have a total of 12 places available on our invitation-only beta programme. If you’re interested in being one of the first people to put KEEP.WORKS through its paces, go to the home page and register, and we’ll be in touch when the beta programme opens in mid-September.
PSC Group and LDC Via have entered into a strategic partnership. Under this partnership PSC Group will adopt LDC Via as a platform for customers seeking to migrate data and applications away from legacy IBM (Lotus) Notes & Domino, and also as a preferred platform for the development of new software applications that might otherwise have been developed on Domino.
Why has PSC Group adopted LDC Via?
Satisfy client needs for a low-risk migration strategy
Create a modern application development platform
Familiar REST-based API model
Familiar security model for Domino applications
Best-of-breed data migration tools
LDC Via provides a cloud-based managed database platform that emulates key features of IBM Domino in order to lower the barrier for transfer of both data and skills. The tiered monthly pricing model of LDC Via, with no minimum commitment, makes it a very cost-effective replacement for Domino. For larger customers bespoke pricing, dedicated installations, and on-premises options, are available: contact LDC Via for more details.
PSC Group is a global professional services and information technology consulting firm headquartered in Chicago.
LDC Via and PSC Group look forward to sharing more about our partnership and the application roadmap we are building together.
We are delighted to report that LDC Via has been awarded a place as a Platform-as-a-Service (PaaS) supplier on G-Cloud 8, the Crown Commercial Service’s public sector procurement framework for cloud-based software and services in the UK.
G-Cloud 8 is the latest stage of the UK Government’s plans to create a digital marketplace; it provides an online catalogue from which public bodies are able to select approved IT services in a simple and cost-effective manner.
Mark Myers, our very own Wookiee, is now in Austin, Texas for the MWLUG conference which kicks off in earnest today. LDC Via is again a sponsor, and has a booth in the product showcase (Four Seasons Ballroom). Come and see Mark, talk about Via, and be sure to check out his presentations too:
SA112: The SSL Problem and How to Deploy SHA2 Certificates (with Gab Davis from The Turtle Partnership) Ballroom D, Thursday 1pm.
IN104: Salesforce for Domino Dogs. San Jacinto West, Thursday 5pm.
Psst… Be sure to ask Mark about our upcoming news!
As well as the technical and feature improvements in the recent “Eton Mess” release of LDC Via, there’s one further change: we have tweaked our pricing model in response to customer feedback. The general picture is that everything has got bigger: so we’ve increased the data volume boundaries at which you move from one price level to another, and with that we have also revised the price bands to reflect the additional capabilities.
“Basic” level - stays £FREE
This is still free, and still has a limit of 500 docs across two NSFs, but the total storage available has gone up from 500Mb to 1Gb. That change allows for testing with lots of large attachments, which is of course not an unusual scenario in Notes/Domino applications.
The storage available on this level has risen from 2Gb to 10Gb. That’s 5x the capacity, for twice the price.
“Pro” level - £499/month (old price £299/month)
The storage available on this level has risen from 10Gb to 25Gb. Which is 2.5x the previous capacity, in exchange for a 67% price increase.
“Enterprise” level - £999/month (old price £749)
The storage available at this level is unchanged at 100Gb, although the price here has increased by 33%. But it still represents a per-storage-Gb saving of almost 50% when compared to the “Standard” or “Pro” levels. And at “Enterprise” level there are no limits on the number of documents stored - the only thing we cap is the total storage space used.
For full details of pricing, and information about our “Dedicated” and on-premises offerings, see our pricing page
Eton Mess was released on June 8th. A full list of all the lovely functionality in this version can be found in our release notes.
A large portion of our time is spent writing applications that use LDC Via as a back-end. As we do more and more development, we identify improvements to the API that will make our lives (and, therefore, the lives of our customers) easier.
In our most recent “Eton Mess” release there have been several changes to the API, but we thought we’d concentrate on just two key updates for now:
The first is a completely new API method dbacl, which returns a list of all people in your organisation who have access to a specific database. By way of example, this method recently proved useful in determining to whom new document notifications should be sent.
The second change we want to highlight is an update to the API methods which allow the storage of unstructured metadata.
Eh? What? Why?
We’re currently writing a new application and we needed to store various settings for the application somewhere, but didn’t really want to create a new database just for this. We can now store those little switches and bits of information about the app in the database details and cache them as required. Handy!
If you want to make use of this new feature, we’ve updated the API documentation to tell you how: set database details.
We’ve said this many times, but it still holds true: we don’t have any hidden APIs. If we discover something that we need then we’ll create a new API and publish it for everyone to use. So, whatever we can do with the LDC Via platform, anyone else can as well. And we’re always looking for ways to improve things: if you have any ideas, please let us know.
Eton Mess was released on June 8th. A full list of all the lovely functionality in this version can be found in our release notes.
Last month we announced the release of “Eton Mess” which includes a number of nifty features and general improvements for the platform. One of the features which Eton Mess brings to the table actually dates back to one of our “Forced Rhubarb” point releases, and the eagle-eyed amongst you may have noticed it in the associated release notes:
Feature: SAML is now an option for organisations wanting to use it. Whoop!
Huge fanfare eh? :-) SAML stands for “Security Assertion Markup Language”, and we use it to provide Single Sign-On (SSO) functionality between an authentication service of your choice, and LDC Via.
Authentication services are known as “Identity Providers” (IdP) in the SAML world. The applications and platforms like LDC Via which take this authentication information and make use of it are known as “Service Providers” (SP).
In practical terms, this means that once the appropriate configuration has happened, an organisation can have its authenticated users proceed directly to LDC Via from their intranet or other corporate systems without a pesky log-in screen getting in the way.
SAML doesn’t mandate the method of authentication used, so Identity Providers just need to support the protocol (as do downstream Service Providers of course). This means that all manner of centralised identity stores can be used as IdPs; for example, Microsoft Active Directory and Google can both act as IdPs.
The good news is that with “Eton Mess”, if you have a SAML environment in mind for integration with LDC Via, we can work with you to make that a reality.
Eton Mess was released on June 8th. A full list of all the lovely functionality in this version can be found in our release notes.
We are delighted to announce the availability of our ‘Eton Mess’ release, as of Wednesday 8th June, nearly two years to the day since we first started creating LDC Via.
In case you’re wondering about the curious naming of our releases, we’ll let you into that secret in a separate blog post. Instead, for now let’s concentrate on a couple of the big headlines from this release.
Performance: the API now allows arrays of multiple documents to be ‘put’ in one call. This allows for much more efficient and performant applications.
Account limits: we have increased the account limits across all (well, all but one) of our tiers. The pricing has been tweaked too, to match. Overall, the new structure seems to fit our customers’ needs better, and that’s what it’s all about.
For a full list of all the lovely goodies we’re sending your way in this release, take a look at our hand-crafted artisanal release notes.
Software developers are always seeking to take advantage of the best of new approaches, tools, techniques and languages. Many come and go, but some have a lengthy lifespan. Domino’s built-in languages of @Formula and LotusScript are pretty obscure these days, mainly because they only exist in the Domino world. Yes, Domino has Java too, but even with that there are some hoops to jump through and some non-standard techniques to grapple with.
Modern environments, in particular the MEAN stack, are taking over as mainstream and very popular tools. Why not take advantage of these skills and opportunities? Moving your Domino data to LDC Via will open it up to all these new worlds, and grant access to a new generation of developers.
Tell us what your organisation’s development strategy is: if we don’t already have sample code for you, we’ll write some.
Back in January we hosted a webinar where we talked about the basics of LDC Via: how to migrate data and use our templates. If you wish to refresh your memory, that’s available up on YouTube: LDC Via Webinar.
Time marches on, and we wanted to put together a second webinar, this time focusing on developers!
Two weeks ago we ran this, Developing web applications with LDC Via, which was loosely-based on some content we first presented at Engage this year. The session was popular, but a lot of people still missed it, and we had some audio issues too. As a result, we repeated the session last Wednesday, and the recording is available on our YouTube channel:
During this session we talk about modern web application development using REST services. There are demonstrations from both the client- and server-side: React.js for some client code consuming the LDC Via API, and also the Java framework Vaadin on the server, together with a node.js / Express application too.
Earlier in the year we hosted a webinar where we talked about the basics of LDC Via: how to migrate data and use our templates. You can watch that video here. But now it’s on to the next topic, and that is going to be developing applications using LDC Via as a back end.
We’re going to talk about modern web application development using REST services. So, we’ll show you demos of developing an application that works with LDC Via data using React, but then we’ll show you how to do exactly the same thing using node.js, and then the same again with Vaadin. We’re going to be talking fast, but hopefully you’re going to learn a lot.
So set the date in your diary: Wednesday April 6th at 4pm UK time. You can sign up here.
We’ve always been keen to make it as easy as possible for you, as a developer, to get authenticated against our API. This is why we used a simple API Key model. We had feedback from some developers that they would prefer us to allow a more traditional session cookie approach to authentication.
In our most recent version of LDC Via, we’ve enabled just that. There are now two different ways that you can authenticate your users against our services. You can continue to use the API Key if that suits your need. But you can also now use a session cookie.
You get allocated the cookie quite simply. You just post the username and password of the user to our service and we return a session cookie if it is valid. Then, whenever you make another request against our services you have to pass the session cookie back to our servers. If the cookie is still valid then we will perform the operation. It’s all documented in our API page.
This is just one example of adding features to LDC Via because you, our users, want them. If you have ideas for improving LDC Via, please do contact us.
The SSL Problem and How to Deploy SHA2 Certificates
Lake Eola A*
Two years ago enabling your site with SSL was a simple affair, buy a certificate or create your own, install it, then just remember to renew it every couple of years. Then, suddenly security holes are being found in SSL virtually every month , popular browsers stop connecting to your site to protect themselves, and you’re continually being told your users data is at risk. In this session we will discuss how it all went wrong and can go wrong again, then go through each step of requesting, generating and deploying a 4096 SHA-2 certificate to use in a keyfile by Domino, IBM Connections, IBM Sametime and other WebSphere products. If you work with these IBM products and need to secure them with confidence this session will show you how!
Matt White will be presenting a quickfire session about node.js:
node.js for Domino Developers
Solution EXPO Theater*
The world of web application development is big, really big. One of the most popular platforms is node.js. XPages developers already have a leg up in understanding node.js and in this session you’ll find out the basics: how to create a basic web application. But then we’ll push it a little bit further; how to integrate an application with your Domino data. We’ll cover the best development tools and some useful tips and tricks to help on your journey to become a node.js developer.
And of course we’re keen to talk to people about LDC Via , so if you’d like to arrange a meeting you can do that here.
We busy folks at LDC Via are hosting a webinar on January 12th which will introduce you to the LDC Via tool and platform. If you want to know what this stuff is all about, and see it in action, you should sign up right away.
Julian and Matt will be presenting the webinar itself, and Ben will be “back-room boy”, responding to any questions on-line and in the webinar chat.
We hope you can join us: we have had a fair bit of interest in LDC Via over the past few months, but equally a lot of people still ask us what it is all about. This session will answer those questions, and more!
We’ve spoken before about LDC Via Lens, but some things bear re-visiting, especially when the LDC Via pixies have been hard at work delivering new features that will put a smile on any developer’s face!
Today, we want to talk to you about “virtual fields”, a relatively new addition to LDC Via Lens. Anyone with a Domino background will be comfortable with the concept of virtual fields: essentially they a modern version of computed-for-display fields in IBM Lotus Notes & Domino applications.
A virtual field forms part of a collection schema, and can be referenced both from an API call and from any user interfaces you use with your migrated data—either within the LDC Via Lens application itself, or your own custom UI.
If you want to see how virtual fields work, and how simple they are to implement, check out our recent video on LDC Via Lens (virtual fields are covered around one minute in, but you should most definitely watch the whole thing).
Domino can be a bit of a silo. In much the same way that the ocean can be a bit wet. We know that Domino is very good as a repository for your data, but what if you need to integrate with other applications and systems? Yes, there are ways of surfacing its data in other environments, but that can quickly get tricky and messy, and doesn’t always perform as well as you might like.
How about this for an idea? Push selected data to LDC Via, with its REST API layer and pleasing user interface, and then integrate with that instead. All modern application platforms play well with RESTful systems, and LDC Via provides a platform for customisation and extension as you see fit. Thereafter, the globe is your mollusc.
We’d love to hear about your Domino integration challenges. Drop us an email, or give us a call.
Fifteen years ago, ‘Lotus Notes’ was the hot ticket. There were countless IT service companies offering software development services to create application after application on Notes. In some cases, when the only tool you have is a hammer, and everything looks like a nail, the applications didn’t pass the test of time. But for many others, the application — probably amended, enhanced and extended way beyond its original concept — still survives as the bedrock of your business.
But… there are no longer countless IT companies offering the right skills. In fact, some might say that the pool of available skilled people — both developers and infrastructure experts — is slowly but surely drying up. The risks of doing nothing are, well, risky.
With LDC Via, you are not restricted to just those skilled with IBM’s Domino Designer client. In fact, you’d be hard pressed to find yourself restricted at all. LDC Via’s comprehensive APIs are all REST-based, so you can pick the development tools and approaches that suit you: .NET, PHP, MEAN stack, Ruby, Python, Java, even XPages — whatever floats your boat.
Liberate your data, and liberate your developers at the same time. LDC Via.
You’re happy and committed to Domino, so you definitely don’t need LDC Via, right? Well, not necessarily.
Domino has always had challenges with scalability. The theoretical/technical limits have increased over the years, and there are all manner of tips ’n’ tricks to make your applications scale. Ultimately, however, there are some limitations in the venerable platform, and it is not the right long-term solution for every application. Or perhaps it was a good choice ten years ago, but with 10x as many users and 100x as much data, it’s struggling.
In fact, the mark of a successful Domino application is one that has lasted years and become overloaded with documents. Why not give the old application a new lease of life? Move historical data off to a platform like LDC Via, where it’s no longer a load on your servers, and let your Domino application’s view indices breathe once more.
Get in touch, and we’ll see what we can do. We think you’ll be pleased.
One of the main uses for LDC Via is in re-homing historical data from servers that are being retired.
Your organisation may find itself in the situation where it is maintaining, supporting, and backing up Domino infrastructure purely so that the user can reference historical data held thereon — no new development, no real activity, few if any data updates … but you still need to “keep the lights on”.
Even worse — and the longer you carry on the more likely this is — these servers may be poorly-supported, may not have robust back-up, and can even be running out-dated or un-patched infrastructure.
So why not kick those boxes into touch and serve up your data in a modern, cost-effective manner (with support, and reliable backups)?
We spoke at a well-attended sponsor session, in part showing off our new offering LDC Via Lens, and had a number of excellent conversations with people. Thanks so much to everyone who dropped by and said hello!
ICON UK raised an amazing £5,000 for The Rainbow Trust this year with attendee fees, a raffle and a gala night auction.
As always with these conferences, there were prizes to be had. Our closing session give-away was a Sonos Play:1, won by Rob Mason of Redtable Information Solutions: well done Rob!
It’s been clear to us for a number of years that many organisations are moving their email off IBM (‘Lotus’) Notes and Domino. Typically they’ll head for either Microsoft Office 365, or Google.
What they are usually then left with is a collection of applications (‘databases’) that were built for the Notes and Domino platform. Some may be Notes client applications, some may be accessed using a browser. Some may be developed internally, and some may be off-the-shelf purchases. What they all require, though, is the retention of the IBM software and servers. And that’s the issue: you thought you were replacing one piece of software with another, but in the end you still need to retain the ‘old’ Notes/Domino servers; and the space they take up in the machine room; and the overhead of backing them up; and the expertise and staffing to maintain and administer them; and let’s not forget the cost of the software licences(!)
So one of the reasons we created LDC Via was to help solve that problem. If you are in that situation, then LDC Via can give you:
a fully managed cloud platform
a robust backup regime
a highly-scalable and very fast cloud-based database engine
a Notes-like security model (‘readers’ and ‘authors’ fields)
point-and-click tools to build a simple UI onto your data
out-of-the-box templates for standard applications (Discussion, TeamRoom, Journal and Mail Archive)
the tools to migrate your data as-is: no complex cleaning or remodelling is needed to shoehorn it into a ‘relational’ structure
comprehensive “APIs” to allow your developers to create new (or re-create existing) fully-featured applications
What we don’t promise is a “magic bullet” solution. There will be some work involved in re-creating your business applications. Possibly quite a lot of work. But there’s a massive ‘plus’ here: using the LDC Via APIs you can do that using any application development framework you choose. So your collaboration applications can now live in the same software development world as your other applications, and no longer be tied to IBM’s “Domino Designer” tool. And your end results are streamlined fit-for-purpose applications your organisation can carry on using for many years.
This year MWLUG comes from Atlanta, GA, promising over forty sessions and workshops on collaboration.
LDC Via will be there! Visit our stand and find out about our latest release, our new offering LDC Via Lens, and our forthcoming developer community, #DevHub.
Dev Hub is a new developer zone coming to LDC Via. This features integrated search with our technical documentation, highlighted blog posts and other information aimed at developers. As a developer you can also sign up for a free LDC Via account using just your email address, and play with the LDC Via API.
So if you’re attending MWLUG, come along to the stand, and we can tell you more!
“Search?” I hear you mutter, “full text search? Really?”
Well yes indeed! We’ve always offered a field-based search facility in LDC Via, but of course, chances are your users will want to search for a phrase across an entire document or collection of documents. Enter LDC Via’s full text search functionality.
Searching like this is often a must-have in an application, but it’s not straightforward to implement when dealing with a platform like LDC Via that offers document-level security. We took some time and did it right, implementing full-text search for those using paid tiers in LDC Via. So if you’re signed up to one of our paid plans, you have a single box to tick and you’re good to go:
Full search functionality will now be available in both the standard templates we offer and the LDC Via API.
You can also combine full text search with standard field-based search in a single query. For example, you could issue a single query to get all contracts with an issue date between 01-Jan-15 and 31-Mar-15 within the category “New client”, and mentioning “banana” anywhere in their text.
As you may know, we already offer free application templates for Discussions, Document Libraries, Teamrooms and mail files. But what do you do when you’ve migrated a custom database to LDC Via and you want to provide your users with a simple view of the data, no bells or whistles?
Enter LDC Via Lens.
Lens is an application template that is driven by configuration, meaning that you can define how a screen of data is laid-out and presented. Using Lens, you can quickly create an interface onto your data that will make sense to your users and allow them to interrogate and explore the data effectively and easily.
LDC Via Lens: key fields
In the Database Viewer admin page, you can elect to “Modify Collection Schema” which shows all of the settings for a collection in your database. What this means is that you can see all of the fields for documents stored in the collection, change their data type, their position on-screen, and whether a particular field is an “LDC Via Lens” field (i.e. available to end users), with an associated field label.
(We’ve added this distinction between “key” and “non-key” fields: all fields are important, but some are not ones that you need or want to expose to end users. By flagging a field as “key”, this means that when you use our Export to PDF API, or Lens, we will just show the key fields.)
In the screenshot shown below, we have defined four key fields. Subject is positioned first, then Categories, From and Body. “From” we want to display with a label of “Created By” and the Body field is rich text, so we’ll just take the automatically parsed HTML version of that field.
Once these settings are saved, select the “Open as Application” link on the Database Viewer page and your work is done:
LDC Via Lens: views
We don’t stop there! If the key field functionality is a nod to half of the old way of doing things in IBM (Lotus) Notes—i.e. forms—then what about the other half, views?
Well, in the Database Settings screen, you can also create view filters. This means that any time a user accesses the database with Lens they can use the “views” that you’ve created to find useful data more easily.
Naturally you can manage your views with our API, so if you’re building your own application, any views you create are still available.
Of course, all of this honours our document-level security model. You are free to distribute application URLs with LDC Via Lens functionality to all your users without any concerns about them seeing data they shouldn’t.
LDC Via Lens allows you to create a read-only user experience for your migrated Notes / Domino data in next to no time, and provides your people with an experience they can easily understand and work with. Let’s go through all this in a video, and you can see exactly how polished an interface LDC Via Lens can give you!:
If you want to go further (i.e. editing content as well as just reading it), we provide a beautifully-crafted set of REST APIs, as well as plenty of code samples to get you started. You can create applications against your precious data (or—of course—brand new applications) rapidly.
We’ve created a full featured API for development of custom applications with LDC Via, however, we’re aware that there is not always the time or budget to do large development projects, so we have also given you a jumpstart.
We’ve worked closely with the team behind the XComponents project. This is an open source framework based on AngularJS and Bootstrap that allows you to quickly create applications that will work on desktop and mobile devices equally well.
All of our standard templates (Discussion, Document Library, Teamroom and Mail) are developed using XComponents, and, even better than that, they’re all open source. You can easily take the source code for them and modify it to fit the needs of your applications.
At the simplest end of the spectrum, you’ll be able to take one of our templates and simply modify the data model to match the forms that you want to display. Of course if you want to get more complicated, then you can extend the templates to do anything that you fancy using the best of breed frameworks like AngularJS and Bootstrap.
There’s no point having an all-singing all-dancing database application, if the data is locked up and can’t easily be extracted, reported on, analysed, and easily re-purposed for use elsewhere. Everybody’s favourite tool for working with business data in these ways is Microsoft Excel. So, naturally, it’s important that LDC Via has great (excel-lent, in fact) support for Excel. And it does, using Microsoft’s ‘Power Query’ feature.
When we were talking to people at the Engage conference in Ghent, this question came up, so we felt it warranted a fairly detailed ‘how-to’ blog post. In fact, the process is pretty simple, and to make it even simpler we’ve not only written it down step-by-step, but also provided a downloadable Excel file to get you up and running.
Using Power Query, which is a free add on to Excel, we can create a parameter-driven spreadsheet that pulls in data from any internet source, including LDC Via.
As ever, the internet is a wonderful source for getting started with these things, so thanks to the likes of Ken Puls and Matt Masson this is what we built in a few hours: LDC Via - Excel Power Query Sample. There are two sheets in the workbook: ‘LDC Via Data’ displays a table of data that has been pulled from LDC Via; and ‘Settings’ contains the configuration settings that define where to get the data from, what API key to use, and which fields to display in the output results.
To create all this we need two Power Query functions and a main “routine” to actually get the data.
The first function is called fnGetParameter and simply reads the settings from the Settings worksheet:
We have four configuration settings:
endpoint defines the LDC Via API endpoint from which data will be extracted. The simplest one will use our collections API that gets lists of documents from a collection
count defines how many documents at a time will be extracted from the API
apikey is your API Key that authenticates you to LDC Via
fields is a comma separated list of field names from the LDC Via collection that will be displayed in the results
Next we need to actually go and get each page of data from LDC Via, and the fnGetCollectionData function performs this task:
The relevant page number is passed in as a parameter called “start”, and then we use the Web.Contents and Json.Document functions to retrieve the data and parse it into an object that we can work with.
The documentation for Power Query, is, shall we say, opaque at best, but there is a list of functions that are available to you on the Microsoft site—the Power Query editor itself feels very much like going back to the Notes @formula editor, but once you get your head around it, you can perform some pretty powerful operations on the data.
In our case we’re loading the JSON document from LDC Via, extracting an array of document objects stored in the “data” property, converting that to a table, and returning all this to the calling function.
Now we need to pull these two functions together and actually run the query to load the data:
Power Query doesn’t really have the concept of a For style loop, so we have to spoof that by creating our own range of page numbers. In this case we’ll loop from 1 to 100, and with 1,000 documents per request that will allow us to get up to 100,000 documents from LDC Via. For each page we call the fnGetCollectionData function, extract the fields that are specified in the settings worksheet, and finally insert them into the main worksheet.
Clearly the speed of all this depends on the number of documents in the database in the first place, but even the example above, with 40,000 documents, takes less than a minute to load and parse all of the data.
To go and get data from a different collection in LDC Via, simply modify the settings worksheet as appropriate, right-click on the GetTableData function, and choose the “Refresh” option: the data will be automatically downloaded and parsed.
Of course, once all your data is loaded you can do all of the things that you’re used to with Excel: pivot tables, filtering, sorting, charts and so on. The world is your mollusc.
We decided that everyone should be able to try out LDC Via for free. With our trial account you get full functionality so that you can see and do everything that paid accounts can. We know there’s a lot to LDC Via, and it can be a little overwhelming, so we thought we’d talk about what you may want to try out having signed up.
First things first, you’ll want to migrate some data. You’ve got two choices here. If your Domino server is internet-facing and you can enable Domino Data Service then a cloud-based migration is the simplest route. Our Migration Cheatsheet document should help you get set up there.
Once your Domino server is configured, you can go to the Database Migrator page and enter the details for your server. You’ll need to enter the URL pointing to your database, your Domino username and password and then select the view that you want to migrate. That’s it! We do the rest of the work for you.
If your Domino server is not internet-facing or DDS is out of the question, then you can use our Desktop Migration Utility. You just need to complete the config file and run the migrator from your desktop machine.
Once done, now that we have some data we’ll want to do something with it. If the database you migrated matches one of our standard templates (Mail, Discussion, Teamroom, Document Library) then you can go to the Database Viewer page and view your application.
The main benefit of LDC Via comes with the use of our REST API of course: you can now create custom applications using whatever technology you prefer. We have samples using XPages, .NET and AngularJS to get you started, but anything that can talk to a REST API can use our services.
Hopefully thoughts will be bubbling to the top of your mind…
What if I want to run a “live archive” of data from Domino so that I can reduce the size of my NSF?
What if I want to securely expose a subset of data behind the firewall to customers via the internet?
What if I want to shut down a Domino server but retain access to all of the data?
It’s traditional at these user groups to run a giveaway competition. Thanks to everyone who entered our word search, you all did very well. The lucky winner of our Sonos Play:1 was Simon Peek from We4IT.
Here at LDC Via Towers we’re all developers, and a large portion of our lives is spent working with badly documented APIs. We’ve just created our own API for you to work with your LDC Via data and we wanted to make sure that it’s as painless for you as possible. Enter the <trumpets sound> LDC Via API Documentation site </trumpets fade>.
The high-water mark of API documentation, in our opinion, is from Stripe and you’ll see that our site is very similar in structure. Our API site is very much a living document. A lot of our time currently is spent using our own API. When we spot things that are confusing or incomplete, we fix them. But as always, if you have any suggestions please let us know using the “Ask Us” link to the side of this post (and indeed on most other LDC Via pages).
Of course, we didn’t develop the site ourselves, we”re too busy writing our own code. Instead we used the wonderful Slate project from Tripit.
MongoDB security concerns have recently hit the press: you may have seen some of the posts. The latest focusses on some German students who identified tens of thousands of MongoDB databases that are wide open: no security on them at all!
We are delighted to announce that the winner of the first and last annual LDC Via ConnectED competition crossword is: Nathan Freeman. Congratulations!
Here is Nathan being presented with his prize before the Closing Session of the conference.
Thank you all for playing our little game. I hope you had as much fun doing the crossword as we had in putting it together. If you’re still wondering what some of the answers were, here is the completed version to put you out of your misery.
Okay, we said we wouldn’t give any more tips, but we’re feeling especially nice today.
If a clue contains a word that might imply things being disassembled or assembled - for example “destroyed”, “in disarray”, “broken”, “rebuilt” - that probably means you’re looking for an anagram in the clue.
Punctuation is important. Where an answer is two words, the letter counts are separated by a comma e.g. [7, 6]. If the punctuation in the letter count is anything else (e.g. [7-6]), pay attention to it. The same applies to punctuation - especially capital letters - in the clue.
Three of us from LDC Via are at ConnectED this week: Ben Poole (@benpoole), Mark Myers (@stickfight) and Julian Woodward (@woowar). We will have LDC Via leaflets to give you, and the crossword and clues are on the back. Do it to the best of your ability (some of the clues are bastard hard - at least we hope so) and hand it back to us by the deadline. Simple. Remember: you could win a Sonos Play:1.
Competition rules, for the avoidance of stress and fisticuffs
Only those physically present at the ConnectED conference venue are eligible for the prize.
You do not have to be a registered conference attendee to be eligible for the prize.
Your completed crossword must be returned to us by midday on Wednesday 28th January.
The prize winner will be drawn at random from the fully-correct submitted crosswords.
In the event of no fully-correct answers, the prize will go to the person (or be drawn at random from the persons) with the most correct answers.
If you are a registered conference attendee, and you cannot locate Ben, Mark or Julian in time, you may email a scan/photo of your completed sheet, by midday on Wednesday, to email@example.com. (Please make sure it’s legible).
The prize will be awarded at the Closing Session on Wednesday afternoon.
If the winner is not at the Closing Session, the prize will be shipped directly to them after the conference.
If you are not at the conference, but fancy having a go anyway (or you are at the conference but want to make a head start) here is the full crossword with clues. We will post a completed version after the conference and you can see how well you did.
We were scratching our heads and twirling our moustaches at LDC Via Towers, wondering what we could do as a bit of fun for ConnectED next week.
And then the penny dropped: what could be more “ConnectED” than … a competition crossword?! At what seems (no official word, but we all know it is don’t we?) to be our last January visit to the Swan & Dolphin, and possibly (perhaps IBM will tell us next week) the last conference recognisable as the “Lotusphere” we know and love, we decided to create a little bit of fun to fill in the dull moments (as if!) and test your knowledge of Lotus, Lotusphere and IBM. And, yes, we’re freely using the ‘L’ word, without embarrassment, because this is an unashamedly nostalgic effort, looking back affectionately on our collective journey down the years with the softwareformerlyknownasLotusNotes and its stablemates from Lotus and IBM.
You may have noticed that this is a ‘competition’ crossword. Yes, there will be a prize: a Sonos Play:1. If you’re not a Sonos user already, you will love it. And if you are a Sonos user already, you will love it.
In the next post we’ll outline the rules for the competition, and reveal the crossword itself.
Meanwhile here, as a little teaser, is the crossword grid.
We’ve talked previously about integrating LDC Via with XPages but the whole point of our REST API is that anything that can talk over a network can use our services. So we’ve been busy.
So far, when you access the LDC Via website, you get some pre-rolled templates that allow you to view your old Domino data in an easier to read way. We provide templates that work with your migrated Mail Files, Discussions, Document Libraries and Teamrooms. But how about creating your own custom applications?
Today we have released our first public template. It’s a standalone implementation of the classic Teamroom that has been created using AngularJS.
What this means is that you can create and deploy quite complex business applications with nothing more than a simple HTTP server, LDC Via takes care of everything else for you including data storage and authentication.
We’ve made the source code for the template available for use from Github under the MIT license, so you will be able to make use of the code to get going with your own custom applications.
Mark, Julian and Ben will all be at IBM ConnectED from Friday evening this week. If you’d like to say hello, talk about LDC Via, enter our competition (come back in a couple of days to find out about that) or simply share a beer / some ludicrous sugary confection (in the case of the Wookiee), we’d be delighted to see you.
As you probably know, the conference itself is taking place in the Swan this year. Mark will be around a lot, although staying off-site, whilst the other two reprobates are also residing in the Swan, so will no doubt be irritatingly easy to find.
Additionally, Mark has somehow become an IBM Champion (no we don’t know how either), and as such you will be able to find him at the Leadership Alliance events looking out of place and desperate to talk tech with anyone.
If you need to know what we look like (oof), take a look-see here. Smooth.
(For those wondering why Matt is not with us this week… he claims to have a very good reason, but we’re unconvinced ;-))
One of the key use cases for LDC Via that we see is for archiving data for successful IBM Domino applications. We all know that Domino has limitations: when a database gets too big, it can really affect performance… but there’s so much good stuff in Domino–like security and integration with email–that it would be a shame to have to migrate your whole application to a completely different platform.
What if there was a “middle ground”? How about moving older or inactive data to a different storage area, but one that still maintains document-level security? You may be surprised to learn how straightforward it is to integrate XPages with other platforms via a REST API.
By way of illustration, we’ve created a sample Domino application that connects to the LDC Via service, obtains a list of LDC Via-supplied databases that the current user can see, and lets the user browse those databases (an individual MongoDB database contains one or more “collections”, and each collection contains documents). Hopefully you can see by looking at the code that makes up the sample app just how easy this is. The main component that drives it all is a managed bean that pulls data from our REST services:
Using this code combined with a single XPage, we can retrieve the aforementioned database list, together with lists of collections within those databases, and for each collection the documents it contains (all based on what the user is permitted to see):
The screenshot shows the list of migrated NSF files (“databases”) that our test user has access to. We’ve selected the database called unpsampler.nsf, and then a collection therein called “Document”. This collection contains 88 documents accessible to the user, and using the API we can then view the first 30 documents, paging through the rest in much the same way you would with a Domino view.
This is a very simple demo, and it wouldn’t take much effort to customise the code to fit your specific requirements. From the end user’s point of view there need be no difference at all in their experience, except that your XPages application is hopefully more responsive as you reduce the amount of live data you’re working with!
There is a phrase regarding security on websites that terrifies developers: “It’s not a question of whether you get hacked, but when.” It’s popular with the media of course―doom-laden articles will always sell, and strictly speaking this terrifying sentence is correct―but only for a given value of ‘correct’.
For example, I don’t think anyone could stop the NSA getting access to private data if the NSA had a thousand years and warehouses of dedicated brute-force script kiddies at their disposal (no doubt backed by law). What we can do however, is make cracking data so bloody hard that the whole process becomes untenable. For LDC Via, we consider the “base data type” for all migrations to be financial transactions from large corporates―in other words, we do not play fast and loose with this stuff.
As we built LDC Via we had to consider the usual suspects in terms of attack vectors, and the defensive layers we need to provide. This is compounded by the fact that IBM Domino has always provided such good, simple security. We provide various options when using LDC Via, from simple data conversion to fully-converted applications with reasonably complex security requirements―and so there are appropriate decisions to be made.
Thankfully, different members of LDC take care of different areas of the application, and this enables us to impose a global security policy with each member taking care of a number of the constituent layers.
We break down the defensive layers into two categories:
“The Usual Suspects”
“Just For Us”
###“The Usual Suspects”###
These are the layers that all web apps have to deal with. Quite a lot of them are irrelevant for on-premises solutions, but take on huge importance in our cloud offering.
Ports and firewalls. An easy one―only those ports absolutely necessary are opened. By default this means SSL (port 443), with the option of port 80 if non-secure transactions are required. That is it: no database ports, no remote access ports etc. We separate dedicated client servers from each other with firewalls as well, so there is no inter-communication between your dedicated boxes and those used by our other clients.
Operating system. We use a hardened Linux build for our production servers, with proper patching and all the things that keep administrators happy.
Hardware. Where provided by our hosting providers we offer hardware encryption. Now that most decent physical storage is based on solid-state technology, we have found that hardware encryption is not the performance horror it once was. If this is a requirement for your implementation, please tell us.
NSA. A new player in the security arena is the National Security Agency in the US. Their curious assertion that they have the right to take any data they deem fit for their purposes from anywhere on the planet means that even if your data is not stored in the continental USA, if the hosting provider has an office registered there then the NSA can pressure that organisation to provide access to your data (with the hosting provider and LDC Via legally obliged not not to tell you about it).
Well, we are registered in the UK, and we have access to multiple hosting providers, both local and international, to meet the different data security needs of clients.
Databases. All databases offer a variety of security options, and the methods we implement will vary according to the back-end server involved. In our current offering (based on MongoDb) we use both standard MongoDb users and roles for security (rather than those wide-open “service accounts” that so often lead to wholesale database leaks in the case of a security breach).
However, that’s not enough: MongoDb only really offers granular security down to a “collection” level, so that leads us on to the the next level of security…
###“Just For Us”
Readers and authors. One of the core tenets of IBM Domino security, and something that is a devil to reproduce in other systems. Readers and authors fields, controlling document- and field-level access to data. Very few systems and databases provide document-level security, so to effect this level of security, we have a “data wrapper”: we have constructed a wrapper that goes round any attempt to access the database from the application (a Java driver wrapper is also provided). The meta-data stored in the database is checked against the requesting user and the various levels of access are granted on that basis. This includes group access as well as individual rights.
Configurable security. At any time, administrators can modify which fields in a collection are used to define security, and of course, will always be able to see all documents (you won’t lose anything!)
So there you go, some of the background to security in LDC Via. As Mark says, it’s like arguing about politics: never-ending, often quite heated, but one day we will build a better world (if the rest of the buggers stop talking rubbish).
We’ve spoken about the whys and wherefores of LDC Via already, but some of the use cases we mentioned in our last post probably bear some expansion. So let’s do that.
1. Re-homing historical data from servers that are being retired
A number of organisations maintain Domino infrastructure purely so that they can reference data held thereon—no new development, no real activity, but they need to “keep the lights on”. These servers may be poorly-supported, may not have robust back-up, and can even be running out-dated or un-patched infrastructure (I think many of us have heard about Domino 6 or 7 boxes still running in production). So why not kick those boxes into touch and serve up your data in a modern, cost-effective manner (with support)?
2. Archiving data from Notes & Domino applications that have grown too large for Domino
There are all manner of tips ’n’ tricks to make your applications scale. Ultimately however, there are some limitations in our venerable platform, and it is not the solution for every hosting use-case.
The mark of a successful Domino application is one that has lasted years and become overloaded with documents. Why not give the old app a new lease of life? Move historical data off to a platform like LDC Via, and let your Domino application’s view indices breathe once more.
(Incidentally, if this use-case is especially of interest, do look out for our LDC Via-XPages integration demo on this very site)
3. Integrating Domino-based data with other applications
Domino can be a bit of a silo. Sure, there are ways of surfacing its data in other environments, but that can get tricky fast. Pushing selected data to LDC Via, with its REST API layer and pleasing user interface could be a solution here. All modern application platforms play well with REST-ful systems, and Via provides a platform for customisation and extension as you see fit.
4. Selectively making Domino-based data available outside the firewall without exposing your Domino infrastructure
5. Migration of Domino applications to other stacks for increased interoperability and performance
Current environments include the MEAN stack, and this has a vibrant user-base. Why not take advantage of these skills and opportunities? This means granting Domino data access to a new generation of developers.
6. Exposing ‘IBM Notes’ application data to non-Notes clients (e.g. web and mobile)
As per item #4, a lot of corporate data may only be accessible via IBM Notes clients inside the firewall. You could Domino-enable such applications in a bid to move away from fat clients. Or perhaps your organisation has a target platform for projects that doesn’t include Domino? Well, LDC Via could be an option.
You’ve heard us talk a little about the Why of LDC Via and hopefully that’s got you interested. But we’re very aware that what we’ve created is a significant change from running your Domino server.
What we need are people to kick the tyres of our NSF migration tools, our admin interfaces, security and also the development API. If you think you’d like to be involved then get in contact with us (some of you already have, thank you!) If you’ve been involved with a beta program before then the process will be familiar, we’ll get you set up with our beta environment, help you get started and then let you play. In return we’ll be asking you to give us feedback, as brutally honest as possible, about the good, the bad and the things that need to be changed before LDC Via is production-ready.
In return for all of your effort you will get our eternal gratitude and very preferential pricing when we switch from beta to live.
Here’s a little more detail about the three main areas of LDC Via we want you to look at:
First we have the migration process. This is split into two parts, either our online tool which will talk directly to your internet facing Domino server, or the on premises tool which acts as a conduit between your internal Domino server and our cloud based servers. We want to be able to migrate your full Domino databases and make sure that we are getting 100% fidelity. We’ve obviously done significant testing of this ourselves already, but we can always do more.
Second we have the admin interfaces. These allow you to see your data. Think of the main admin screen as the equivalent of an “All Documents” view combined with the “Document Properties” dialog box (only more useful, hopefully!) You can manage user accounts and security for your organisation too, which includes modifying document-level security settings.
Finally, for the developers out there, we have our API. This is what allows you to create new applications or integrate with your existing applications and access your data in whatever way you need. Here we’re wanting to make sure that the APIs we have created are sufficient, and to identify new APIs that will make your life easier.
So if all this sounds like it’s something you want to get involved with, please let us know and we’ll add you to the list. We aim to be starting the process in the next few weeks.
In part one of these posts we took a look at why we created Via. In this post let’s explore what that means for you.
First, let’s revisit what Via is, in a nutshell:
A modern hugely-scalable and highly-performant database platform
A secure application layer that gives you all the Readers / Authors / Roles / Groups goodness you’ve come to love in Domino
A set of tools to migrate Domino data into Via
Sound good? So where does that fit in your plans? Here’s where we see Via as a viable (see what we did there?) future for your Notes and Domino databases:
Re-homing historical data from servers that are being retired
Archiving data from Notes/Domino applications that have grown too large for Domino
Integrating Domino-based data with other applications
Selectively making Domino-based data available outside the firewall without exposing your Domino infrastructure
Migration of Domino applications to the MEAN stack for increased interoperability and performance
Exposing ‘Lotus Notes’ application data to non-Notes client (e.g. web and mobile)
The list is endless. Well, long anyway.
The NSF data structure is a truly excellent piece of software engineering for its time. And it was way ahead of its time. But the rest of the world has finally caught up and, arguably, overtaken. Except in the area of security, where NSF’s Readers / Authors security model is still unique in its power and elegance. Or, at least, it was until now: LDC Via.
We thought it would be helpful to explain why we’ve created LDC Via, and why you might want to use it.
Between us at LDC we have about 70 years of experience working with IBM (Lotus) Notes and Domino. We’ve been there through all the ups and downs and ins and outs of the product’s history since version 3 back some time in the late 20th century (when you were a mere lad or lass, of course).
What we’re seeing now is that Notes and Domino are viewed (whether correctly or not—that’s a separate discussion!) as being somewhat old-fangled, uncool and—that horrible word—“legacy”. The upshot is that some organisations are reducing, or even stopping, investment in this platform. For email they may well be migrating, or have migrated, to Microsoft or Google. Those that are staying with IBM may be considering moving to IBM SmartCloud for email. So, for many, the on-premises Domino servers have become a liability, a problem, and an unwanted overhead. But—and it’s a big but—there are important business records and data locked away on those servers.
So that’s the first reason why we built LDC Via. We believe that many organisations need a route away from Domino as a strategic data store. That may not mean they’re dropping Notes / Domino entirely, of course. But it may mean that they want to downscale or eliminate internal infrastructure, it may mean that they want to retire the Notes / Domino applications but have the data available, or it may mean that they want to redevelop the applications on an alternative platform.
Alongside this gradual shift in focus away from Domino, has been a recent and fairly rapid rise in “NoSQL” databases. The irony, of course, is that Domino’s NSF data structure is itself a NoSQL database. But it’s an ageing one, with undoubted issues of performance and scalability when compared to some of the younger contenders. And with the rise of these databases has been a parallel rise in acceptance of new development architectures to make the most of them, most notably the “MEAN stack”. Here at LDC Via we opted to develop first against the MongoDb back-end, as that is arguably the most accepted and widespread of these modern NoSQL databases.
Where the Domino NSF database has always been a market-leader is in its security structures. Readers fields, Authors fields, encryption keys, database encryption, etc. There just isn’t anything else like it. Except, well, now there is: LDC Via.
We hope that’s given you a glimpse into why we’ve built this product. It’s because we believe you need it. In part two we’ll explore why we believe that.
Welcome to a new offering from LDC. We call it LDC Via.
Over successive posts we’d like to introduce you to this new tool, designed to liberate your information from older IBM Domino (Lotus Notes) data stores.
Perhaps your organisation is moving away from Domino, or simply no longer investing in maintaining the platform. Maybe you have applications in need of an overhaul, or systems are being retired, but you want to retain your data…
As the name suggests, LDC Via represents a path, a route away from Domino, whilst retaining the things you need: document-based data, name-, group- and role-based security, and more.
In short, Via brings delightfully simple access to your information, whether it’s on premises or in the cloud.
LDC Via: modernise your Domino data.
Use the cloud to liberate your organisation’s knowledge.