J.D. Marymee
You think your company has a large Novell Directory Services (NDS) tree? I bet your company's NDS tree doesn't contain one million User objects. You may even wonder if such an NDS tree is possible.
Last year several Novell employees and I proved that an NDS tree could, in fact, contain one million User objects. Unbeknownst to Novell managers, we spent two days in Novell's SuperLab populating an ordinary NDS tree with one million User objects. At the end of our covert operation, the SuperLab had the largest NDS tree Novell had ever seen. This article explains how we designed, created, and populated the NDS tree.
As you might imagine, populating an NDS tree with one million User objects requires careful planning. We based the design of the NDS tree on standard NDS guidelines:
To ensure that the NDS tree could be duplicated by other companies, we used standard hardware and software. We loaded intraNetWare and intraNetWare Support Pack 3.0 on each server, which had a 233 MHz Pentium processor, a 3-GB hard drive, and 64 MB of RAM. (Novell has since updated intraNetWare Support Pack. You can download the latest version, intraNetWare Support Pack 4.0, from Novell's Support Connection World-Wide Web site at http://support.novell.com.)
In addition, we installed NetWare/IP 2.2 (which is included with intraNetWare) on each server, creating a TCP/IP network capable of running IP-only routers and accessing the Internet. We then connected the servers using 100 Mbit/s Ethernet links.
The entire network was comprised of 27 intraNetWare servers split across three racks with five workstations attached. (See Figure 1.) Each rack held nine servers, eight of which held partitions and replicas. The ninth server on each rack was called the infrastructure server, which was designed to track network trends and monitor network traffic as NDS synchronized partitions and replicas.
To monitor network traffic, we installed Novell's ManageWise on each infrastructure server. In this way, we could determine how much network traffic NDS generated and how this traffic would affect similarly large NDS trees.
After we connected all of the servers, we created the NDS tree. We placed a replica of the [Root] partition on each infrastructure server. (We placed the master replica of the [Root] partition on the infrastructure server on rack one, and we placed a read-write replica of the [Root] partition on both infrastructure servers on racks two and three.) Because we planned to partition the NDS tree so that the [Root] partition contained few NDS objects, each infrastructure server could hold a replica of the [Root] partition and still be able to monitor the NDS creation and replication processes without participating in these processes. At this point, the NDS tree was fairly basic.
To understand the philosophy behind the partitioning scheme we used, you have to understand how NDS works. When you are authenticated to an NDS tree, the Novell client software on your workstation connects to a particular server. If you want to create a User object in a partition that is not stored on the server, the Novell client software must then connect to the server that does contain a replica of the partition.
For example, suppose that an NDS tree contained three partitions: The [Root] partition and the San Diego partition were stored on the SD_SERVER server, and the Salt Lake partition was stored on the SLC_SERVER server. If you were connected to the SD_SERVER server and tried to create a User object in the Salt Lake partition, the Novell client software on your workstation would have to connect to the SLC_SERVER server since that is the only server on which the Salt Lake partition exists.
When the one million User objects were ultimately created, we wanted these creations to occur on each server, eliminating the network traffic between master replicas. To create an NDS tree with one million User objects, we had to be able to create User objects from multiple workstations.
We created eight container objects in the [Root] object. (See Figure 2.) In each of these container objects, we created 10 additional container objects, in which we created another 10 container objects. Finally, we created 1,250 User objects in each of the bottom container objects. (The next section explains how we created these User objects.)
We made each container object a separate partition. Although making each container object a partition wasn't necessary, we partitioned the NDS tree in this way so that we could create even more User objects if necessary. We then placed each of the eight parent partitions and its subordinate partitions on a separate server.
Each server, therefore, held 100 NDS contexts, each of which contained 1,250 User objects. (See Figure 2.) 100 times 1,250 equals 125,000 User objects per server. 125,000 User objects times eight servers equals one million User objects.
Why did we create four layers in the NDS tree? Let's analyze the design of this tree based on Novell's guidelines for creating NDS trees:
But what about synchronization traffic between replicas? The following points explain how much synchronization traffic would occur:
Because manually creating one million User objects was out of the question, we had to devise an automated process to create these User objects. We explored several options:
After testing each option, we chose the NLM option. This NLM was, in fact, the only customized code created for the project. We decided to use an NLM rather than a Windows-based application for the following reasons:
After creating the partitions and replicas, we disconnected the second and third racks from the backbone, leaving only the servers in the first rack running. We then loaded the User Creation NLM on each server in this rack. In this way, we could create one million User objects and connect the other two racks to the backbone again to gauge synchronization traffic.
The time finally was at hand to create the User objects. Twenty-five hours after we ran the User Creation NLM, this NLM had created 125,000 User objects on each of the eight servers.
At this point, a catastrophic failure on any of these servers would have forced us to start over, so we immediately began the synchronization process to provide fault tolerance. We connected the other two racks to the backbone and used the infrastructure server on each rack to make the following observations:
At last, the NDS tree contained one million User objects. We then randomly selected User objects and logged in to the network to check the validity of these User objects. Not a single login failed.
Each server volume occupied approximately 350 MB of hard drive space. As you might expect, a fair amount of hard drive activity occurred, but background synchronization traffic was virtually nonexistent. When we created a User object, for example, NDS replicated this object only to the servers on the other racks.
How long did NDS take to search for a User object? Well, searching one million User objects does take some time. As an experiment, we added the location of Hawaii to a random User object. We then used Novell's NetWare Administrator (NWADMIN) utility to search for a User object with Hawaii as its location. We received an answer 23 minutes later, which is not bad considering that we were performing a linear search across a distributed database. However, this time is not acceptable, even for a large NDS tree.
Fortunately, Novell is working on a solution that may speed up the process. Called catalog services, this solution will be available in NetWare 5. As the name implies, catalog services uses a directory catalog, which maintains an index of objects in the NDS tree that network administrators and users are interested in. For example, you could catalog all User objects or all User objects with a last name that begins with an M. A directory catalog also forms the basis for a contextless login. (A future issue of NetWare Connection will provide more information about Novell's catalog services.)
As an exercise in adventure, we decided to go even further. A major telecommunications vendor (who shall remain nameless) challenged us to prove that an NDS tree could contain two million User objects. So we decided to double the size of our NDS tree. We added 1,250 User objects to each of the 10 lower container objects for a total of 2,500 User objects. Each server then held 250,000 User objects. As a result, we had created an NDS tree with two million User objects.
As luck would have it, several server failures occurred during the month that our NDS tree was in operation. At one point, two servers holding read-write replicas of the O=Two partition failed, but no one was the wiser--until we discovered the smoking remains of these servers! The NDS tree remained fully functional in spite of the server failures because NDS is a distributed, replicated X.500-based directory.
Scaling NDS to hold one million User objects isn't a trivial task, but it can be done with proper planning. Of course, you should have realistic expectations: Do expect to use hard drive space to store the User objects, and don't expect extremely fast searches without the upcoming catalog services. Also, you should properly place replicas for the best performance and the most efficient replication.
NDS is an excellent database for directory operations, no matter what size of NDS tree your company requires. In fact, Novell is even working on some solutions for companies that want to store a million User objects on one server (although I have never seen a million people in one building). These solutions are targeted for use by companies, such as Internet service providers (ISPs), that support a huge number of users.
In closing, I have to say that NDS scaled far better as a true distributed directory than I ever imagined. So like us, don't be afraid to push the envelope as long as you're willing to plan properly.
J.D. Marymee works for Novell Inc. in Provo, Utah.
NetWare Connection, April 1998, pp.38-40