Performance

Kii Cloud is carefully designed and implemented so that it will scale with a ton of users. When application data are overpopulated in a certain bucket, however, you might experience some performance degradation. This page discusses the factors that will affect the performance and some techniques you can take to alleviate the performance degradation.

This topic describes considerations for cases where a user or a group holds hundreds of thousands of KiiObjects. If a user or a group holds just up to a few hundred KiiObjects, you do not need to particularly consider for performance except for the issues described in Data Transfer below.

Number of users

Kii Cloud is designed to scale automatically if the number of users increases. The increase in the number of users does not cause any performance degradation.

Be careful, however, that the number of objects in a certain bucket might increase as the number of users increases (especially when you are using an application and group scope bucket). If every user is writing one object in an application scope bucket, for example, this bucket will have one million objects if there are one million users. In such a case, the performance deterioration could occur if you do not carefully design how you store objects in buckets. Check the next section for more discussion.

Bucket and KiiObject design

The fundamental tip for preserving the performance is to maximize the number of buckets and minimize the number of objects in each bucket. Having many buckets in an application does not affect the performance, and if you can keep the number of objects in each bucket small, you should not need to worry about any performance degradation.

This section covers some techniques you can take to preserve the performance when you cannot follow this fundamental tip (i.e., you can't avoid putting many objects in one bucket).

Splitting a bucket according to the usage

If possible, try to split a bucket into multiple buckets based on how the objects are used.

For example, suppose that your application publishes online articles. If the most of the users will just browse the latest articles, you can create a dedicated bucket for storing only the latest articles. This will ensure the performance of querying and fetching the latest articles because the corresponding bucket will always have the limited amount of data (see the figure below).

Only the drawback is that you will now need to query two bucket when you are searching an article from both the latest and previous articles. Still, the overall performance should improve if such the query is not the common usage.

Nesting KiiObjects

You might consider "nesting" objects to reduce the number of objects. For example, you can set an array for storing multiple objects in one object.

Suppose that your application is handling sensor data that are generated continuously. By nesting, you can aggregate multiple sensor data in one object. This will reduce the number of objects you need to handle, and also reduce the communication traffic and the number of API calls.

One downside of the object nesting strategy is that the querying and editing of data become difficult. Still, it is worthwhile to consider this strategy if the users are mainly viewing the data. Determine if this strategy is appropriate for your application by considering the factors like the application specification, the estimation of the amount of data generated, and how and how much the data are accessed.

Query performance

Kii Cloud automatically creates indexes for providing quick object search even if there are many objects. For some queries, however, the indexes do not work effectively, and thus the performance is affected by the number of objects.

If you are going to query for more than a hundred thousand objects, you should keep in mind the following characteristics:

• Querying with simple conditions concatenated with "And" will work quite efficiently. The performance is usually better than the linear to the number of objects.

• Retrieving one object with its ID and URI (e.g. the CreateByUri) will be affected little by the number of objects in the bucket.

  • Querying with the "Equal (EqualClause)" usually will be affected little by the number of objects. However, if the target field has a low cardinality (i.e., the field has few unique values as compared to the number of objects) like gender, the query performance will be affected by the number of objects.

• Querying with the "Or" or "Not (NotEqual)" clauses will be greatly affected by the number of objects. Try to use other query conditions if possible.

  • Consider transforming your query conditions to avoid using the "Or" clause. For example, if you want to query with the conditions (key1 = a And key2 = b) Or (key1 = c And key2 = b), you can transform them to (key1 In (a, c) And key2 = b) by aggregating with the key2. This will improve the query performance.
  • Using the "InClause (InWithXxx) will have better performance than using the combination of the "Or" and "Equal".
  • Consider using the "InClause (InWithXxx) if the target field value has limited variation. If the field only takes an integer from 1 to 5, for example, you will gain the performance by using the InWithIntValue 1, 3, 4, 5 instead of the NotEqual 2.
  • You can often transform a Not(expression) clause to an equivalent clause without the Not operator. Consider whether you can transform the clause as stated below.

• The Object ID has a role of the primary key index. If you have a field that will be frequently updated and know that its field value is unique in the bucket, you will gain the performance by assigning the value as the Object ID.

• The performance of Geo query varies depending on the data. It is not easy to estimate the performance tendency, so we recommend you to evaluate the performance beforehand with your data.

  • Kii recommends narrowing search results with a method other than Geo query if the number of queried objects is large. For example, if you use a technique such as Geohash, query performance might be higher.

• The following queries might cause some bad effects on the performance:

  • Deeply nested queries.
  • Complex queries.
  • Querying with only a field with low cardinality (e.g., filtering with a boolean field).

Transforming a not clause

When a Not(expression) clause seems necessary for your query, you can often transform it to an equivalent clause without the Not operator according to De-Morgan's laws. You can prevent poor performance by transforming the clause to that without the Not operator.

De-Morgan's Laws:

• Not( A Or B ) = Not( A ) And Not( B )

• Not( A And B ) = Not( A ) Or Not( B )

For example, suppose objects have a numeric field Age and a numeric field Answer that can only hold a numeric value from 1 to 5. If you want to retrieve objects which do not meet the condition of "Age is less than or equal to 40 or Answer is 2 or 3", you can transform the clause to the one without the Not operator.

Not( ( Age <= 40 ) Or In( Answer, [2, 3] ) )
= Not( Age <= 40 ) And Not( In( Answer, [2, 3] ) )
= ( Age > 40 ) And In( Answer, [1, 4, 5] )

If the transformed clause still contains Or, there is a possibility of poor performance. Consider whether you can further transform the clause by means such as those mentioned in Query Performance.

Improving sort performance

If you are going to sort the query results, you can improve the performance by including the sort key in the object query conditions.

For example, suppose that you want to list up the query results while sorting them based on the score. Inserting a dummy query condition that contains the sort key and matches with all data (e.g., "the score is greater than or equal to 0") will speed up the sorting.

Data transfer

If clients simultaneously make many API calls, it leads to increased processing times and server loads.

Pay attention to the number of API calls if multiple KiiObjects are accessed. The pace of increase in the processing time and the number of API calls varies depending on the characteristic of the used API.

• KiiObjects are processed one by one when they are written and when they are read by their ID. If the number of KiiObjects is high, it leads to increased processing times and many API calls. Processing just a few tens of KiiObjects via a public line can take more than ten seconds.

• When KiiObjects are read by a query, one API call gets up to 200 KiiObjects. Therefore, the number of KiiObjects has relatively less impact on performance.

The Kii Balance tutorial describes optimization of data transfer. For more information, see Performance of Data Access.

In order to mitigate server load of querying large amounts of data, it is more appropriate to read data sets in order as they are required on the screen than to read all data at once.

Distributing server loads

If you need to transfer many KiiObjects one by one, consider distributing server loads. Design your mobile app to prevent a high number of data accesses, especially when active users simultaneously access Kii Cloud because of a marketing notice or a release of new features through media or push notification.

Kii Cloud returns an error if there are accesses that greatly exceed the ordinary load to the server within a certain period of time for an application. The limit is high enough for ordinary changes in the operational load. However, if active users simultaneously send a request, it could cause an error.

If a high number of data accesses is expected, consider the following measures.

• Nest data items as many as possible in a KiiObject with the method described in Nesting Objects above. It reduces the number of transfers, loads, and the processing time.

• Distribute server loads and avoid troubles by having the clients gradually read or write in the background and using random numbers that are unique across devices to set different start times for a certain task.

Other techniques

Using server code

If there is a lot of communication between the client and Kii Cloud, you could improve the performance by using the server code. The client can delegate the necessary execution to the server code, and it can just get the result.

The approach works because the communication among servers in Kii Cloud is much faster than the communication between the client and Kii Cloud. As presented in Transactions, this approach also has an advantage of preserving the data consistency. Note that the internal API calls made in the server code are counted in this approach (i.e., you won't save the API calls).

Bulk-deleting KiiObjects

Rather than deleting objects one by one, it is quicker to delete all objects in a bucket by deleting the bucket (if applicable). This approach will reduce not only the processing time but also save the communication traffic and API calls.

Querying as the app administrator

You will gain the performance if you query a bucket or count objects in a bucket as an app administrator.

When a normal user starts the object querying or counting, the process first determines which objects in the bucket are accessible by the user. Then, the process queries or counts only these objects. When the app administrator starts the object querying or counting, this ACL check is skipped, and all KiiObjects in the bucket are queried or counted instantly. Of course, the results will be different if there are some objects that cannot be accessed by the user (so be careful).

The performance gain will be bigger on the object counting. This is because the weight of the ACL confirmation process is bigger in the counting process.

One way to implement this technique is writing server code that queries or counts the objects as the app administrator and executing this server code manually.

We recommend you to use this technique if the target bucket contains more than hundred thousand objects.