Hibernating Rhinos

Zero friction databases

Feature explained–Spatial Geocode from address

In August we added the option in spatial queries to get the Geocode by address.

We already did something like that in the “Events” page in the RavenDB website and It was quite easy with the Google Maps API.

However when we tried to implement it in the studio we discovered that to do that in Silverlight is not as simple.

Apparently in order to get Google API to work with Silverlight you need to set a bridge server that has the appropriate clientaccesspolicy.xml because the settings on the Google API server does not match the permissions required for Silverlight.

So we looked for another service that has the settings that allowed to work with Silverlight without any complications, what we found is that the Yahoo! Maps Geocoding does just that.

The setting of the Yahoo API was even easier then Google’s and is done like this:

   1: var url = "http://where.yahooapis.com/geocode?flags=JC&q=" + queryModel.Address;
   2:             var webRequest = WebRequest.Create(new Uri(url, UriKind.Absolute));
   3:             webRequest.GetResponseAsync().ContinueOnSuccessInTheUIThread(doc =>
   4:             {
   5:                 RavenJObject jsonData;
   6:                 using (var stream = doc.GetResponseStream())
   7:                 {
   8:                     jsonData = RavenJObject.Load(new JsonTextReader(new StreamReader(stream)));
   9:                 }
  10:  
  11:                 var result = jsonData["ResultSet"].SelectToken("Results").Values().FirstOrDefault();
  12:  
  13:                 if (result != null)
  14:                 {
  15:                     queryModel.Latitude = double.Parse(result.Value<string>("latitude"));
  16:                     queryModel.Longitude = double.Parse(result.Value<string>("longitude"));
  17:                 }
  18:  
  19:             }).Catch();
  20:  

As you can see it is simply a web request with a default url with the addition of the address

For mode information about the Yahoo geocode API: http://developer.yahoo.com/maps/rest/V1/geocode.html

New Option in the RavenDB Studio–Patching

Not long ago we added the option to do Eval Patching in RavenDB (see: http://ayende.com/blog/157185/awesome-ravendb-feature-of-the-day-evil-patching)

Recently we added this option to the studio.

Now when you open the studio you will see a new tab in the options for “Patch”

image

Once there you will see the next screen:

image

 

As you can see this page has many options so lets look at them separately

image

At the top left you have this toolbar with several options, the first one is a combo-box where you can choose what you want to patch (“Document”, “Collection” or “Index”).

Next to it you have the “Test” Option, with this option (which will only be available after selecting a document to check the patch on) when we click on this button the fields on the bottom of the screen (titled “Before Patch” and “After Patch”) will be populated with the selected document and how will it be after we patch.

Next we have the “Save” and “Load” options. With those you can save this page setting and load them again for later use.

Next we have the Patch option that executes the Patch command (this will save the changes to the database)

Let’s look at a small example of patching:

In this case I want to change the Genre name from “Rock” to “Rock and Roll”, after I pressed “Test” I got the screen below, as you can see the “Name” is different.

image

as you can see on the right of the script we have the “Parameters” list, in there I can do this patch in a different way:

image

Now I as you can see we added another value to the document with the help of parameters.

 

Now lets go over the changes when using “Collection”

image

In “Collection” you can see that the “Patch” option changed into 2 options: “Patch Selected” and “Patch All”.

In order to patch only some of the items in the collection you can select then in the “Matching Documents”  section that was added to the page.

Another thing we have is the option to select on which collection we want to patch.

 

And last we have the option to patch by index:

image

In addition to the changes we had for “Collection” we now have a field “Query” for the index and the patch will apply only to the documents that match this query (and the “Matching Documents” will update accordingly).

Update in RavenDB: A new statistics page

Lately we updated the statistics page in RavenDB Studio.

This is how the new page looks:

clip_image002

If you look you can see that the data we see in "Artists" is different than what we see in "AlbumsByCountSold" this is because "Artists" has data on reduce and we want to see that.

Another thing we added is the option to view statistics in groups, In the top left you can see a ComboBox in there you can choose from several options:

clip_image003

If you select "Single Items" you will see all the data that is not a part of a list:

clip_image004

For each index in your database you will see an item in the ComboBox that will show the stats for this index

clip_image006

In addition you can view all the indexes.

If you have stale indexes another section will be added to the page:

clip_image008

If you'll press the stale index name you will see the stats for this index.

Another thing we have added is that in the query page if you are using an index (not dynamic query) you can go directly to this index statistics

clip_image010

The most right button will send you to the statistics page with the data for this index displayed.

Published at

Originally posted at

New feature in RavenDB Studio: get spatial geocode from address

For a while now we had the option to do spatial search, in order to do that we need the longitude and latitude of center we want to search around.

However most of the time that we really want to check is result round a specific location (we don’t really care about the longitude and latitude but the address they represent) so usually you would look in some site that you input an address and you get the longitude and latitude.

With the new feature in RavenDB Studio we let you enter an address and we update the longitude and latitude for you:

The new UI:

clip_image002

We enter an address:

clip_image004

And after we press the "Calculate from address":

clip_image006

It is that simple.

In my next post, I’ll discuss exactly how we did it. In turned out that it isn’t as trivial in Silverlight as we might want.

Published at

Originally posted at

Dealing with conflicts in RavenDB studio

Recently we added the option to manage bundles in the studio (see http://blogs.hibernatingrhinos.com/12577/ravendb-1-2-studio-features-the-new-database-wizard).

One of these bundles is the Replication bundles, sometimes with replication a conflict can happen (this could happen by several reason like 2 or more documents with the same id being created on different destinations), the way RavenDB deals with this is by creating one document to each of the versions available of the document and in the “original” document we have the names of these copies.

With the new conflict resolution UI when you try to edit the original document you will receive a suggestion of resolution.

Let create a conflict on purpose and see some of the options:

In order to create the conflict we will create 2 databases on the same server, both with replication bundle enabled but we will not set the destination yet:

clip_image002

Now we will create a document on each database with the same Id:

 

clip_image004 clip_image006

Now we will edit the destinations on the left and connect it to the right :

clip_image008

Now we will go to the destination and look the the documents list:

clip_image010

As you can see we have 3 documents with Id starts with “Example/1”.

Now we get to the interesting part if we go and try to edit “Example/1” we will get the next page:

clip_image012

The Studio realizes that we wanted to edit a document that had a conflict and gave us a suggestion, as long as there are comments on the document you will not be able to save it.

Now you can see that RavenDB Studio saw an issue with the Name, we give you a list of all possible names and all you need to do is delete what you don’t want there.

Let say we want it to be “Hibernating” we will delete what we don’t want and save:

Before save:

clip_image014

After save:

clip_image016

Now if we take a look at the documents list you can see that we are left with just the fixed version:

clip_image018

Now let look what RavenDB will suggest for several other conflicts:

Example 2 – Same Item:

clip_image020 clip_image021

The result:

clip_image023

Notice that we still let you know that a conflict has accrued but you have not comments in there which means you can simply save to resolve the conflict

Example 3 – List With Different Items

clip_image025 clip_image027

The result:

clip_image029

We put both items in the array but add the comment so that the user will know that the data was changed.

Example 4 – List with some items the same and some different

clip_image031 clip_image033

The result:

clip_image035

We combined all the data into one array with no repeats and inform on the change.

For more about conflicts read: https://ravendb.net/docs/server/scaling-out/replication/handling-conflicts

Creating a Wizard in Silverlight

When creating the new “Create New Database” for the studio we ran into an issue with showing several child windows one after the other (and keeping in mind that if in one page cancel was click we don’t want to show the next windows) like an install wizard.

Unfortunately Silverlight does not have anything like that so we needed to create on for ourselves.

First thing we had in mind when we came to approach this is that Silverlight works asynchronously which means that we want to wait for a page to close before we open the next one.

The way we dealt with that was listening to every windows Closed event and on close open the next page (if exists) the Wizard class looks like this:

clip_image002

The ShowNext() method will check to see if we have more windows to show and show the next one or return with the result if we are done.

We use a TaskCompletionSource and the TPL to make sure that all of that is wrapped in a nice Task, and that is pretty much it. With that done, we can start using it. You can see how easy it is here:

clip_image004

Published at

Originally posted at

Dynamic Tab View In Silverlight

With the new “Create New Database” feature to the RanvenDB Studio we added the option to edit your bundles.

We wanted something that seemed to be quite strait forward: A tab view that we could dynamically choose which tabs a visible and which are not.

At first we tried doing it with the TabControl and bind the header visibility of each TabItem to a property in the model that let us know if this database has that bundle.

This is how I had done it:

clip_image002

At first it seemed to work, however when looking closely we noticed something like this:

clip_image004

clip_image006

As you can see, in the header we only have “Versioning” but the tab we see is the “Quotas Tab”

This happened because we hid the header but the item itself was still the “SelectedItem”.

So we tried to bind the “SelectedItem” as well, but apperantly there is an issue with TabControl and MVVM.

Finally we decided to create our own implementation of Tab View and this is how we did it:

clip_image008

In the Model size we have:

clip_image010

The Bundles Collection is updated with the list of bundles that are active for this database.

One other this we needed to add was to initialize the SelectedBundle to be the first one in the list.
We did that where we initialize the Bundles collection like that:

clip_image012

RavenDB 1.2 Studio Features: The New Database Wizard

In the new version of the RavenDB Management Studio we wanted to enhance the options you have, one of these enhancements is the new "Create New Database" wizard.

We wanted to add the option to choose bundles for your database, configure various options and in general give you a easy way to control options that you could previously do only if you mastered the zen of RavenDB.

Now then you select the New Database option from the databases page you will get the next window:

clip_image002

If you check the “Advanced Settings” option you will get this window:

clip_image004

Now you can change the location where the data (path), logs and indexes will be stored.
In some cases putting the data, logs and the indexes in different drives can improve performance.

On the left side you have the Bundles Selection area, every bundle that you select here will be added to your database. More data about bundles: https://ravendb.net/docs/server/bundles

Some of the bundles have settings that you can (and should) configure:

Encryption:

clip_image006

In here you can choose you encryption key (or use the random generated on) after the database creation a pop up with the key will show you the key on last time. Make sure you have this key saved somewhere; after that window will close we will not give you the key again.

Quotas Bundle

clip_image008

In here you can set the limits for quotas (see http://ravendb.net/docs/server/bundles/quotas)

Replication Bundle

clip_image010

In here you can set the replications (and add as many as needed)
(see http://ravendb.net/docs/server/scaling-out/replication)

Versioning

clip_image012

In here you can set the versioning (see http://ravendb.net/docs/server/bundles/versioning)

Notice that you can’t remove the default configuration or change the Collection for it but you can edit the values.

After you will press “OK” the database will be created (if you have the encryption bundles you will now see the encryption key one last time)

If in the future you want to edit the setting to the bundles (for quotas, replication and versioning) you can right-click the database in the databases view and select “edit bundles”

clip_image014

Note: you cannot change which bundles are part of the database after creation.

Published at

Originally posted at

Implementing a VirtualizingWrapPanel

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

If you’ve used Silverlight or WPF for any length of time, you will know all about Panels – StackPanel, DockPanel, Grid and the like - which handle the positioning of UI elements on screen. You’ll also know about VirtualizingStackPanel, which can work wonders for UI performance when displaying substantial lists of data. It does this by deferring the creation of UI elements for off-screen list items until they are scrolled into view. (If you don’t know about Panels, and want to learn, a good place to start is on Dr WPF’s blog)

What if you have a huge stash of data but don’t want to display it in a boring old list? What if you want items laid out like cards on a table? Silverlight and WPF both include a WrapPanel class which can achieve the desired layout. But as its name suggests, it doesn’t virtualize. Use a WrapPanel in your 10,000-item ListBox, and you’ll be twiddling your thumbs whilst it goes off to create elements you might never see.

This was a problem we faced when we were implementing the new data virtualization features in RavenDb Studio 1.2, which lets you scroll through your entire collection of documents. We wanted the card layout, but WrapPanel wasn’t up to the job of working with huge numbers of items.

So I set about implementing a VirtualizingWrapPanel, which I’ll share with you today. You can see it in action in our sample Netflix browser (switch the Display Style to Card). The code is all available on GitHub.

First I’ll give you a quick refresher on how you make ListBox use a custom Panel, then I’ll run through how one goes about implementing a virtualizing panel in Silverlight.

A Quick Refresher on Using Panels

Here’s a snippet of XAML showing a ListBox using a VirtualizingWrapPanel:

<ListBox x:Name="DocumentsList"
                        ItemsSource="{Binding Items}" DisplayMember="Item.Name">
      <ListBox.Template>
          <ControlTemplate>
              <Border CornerRadius="2" 
                    BorderBrush="{TemplateBinding BorderBrush}"
                    BorderThickness="{TemplateBinding BorderThickness}">
                  <!-- We set the TabNavigation to Cycle on this ScrollViewer to work around a bug which causes the ListBox to loose focus when navigating down (with Down Arrow or Page Down) from the last visible item
              (or even when navigating Up/Down on an item that is only partially visible at the bottom of the screen) -->
                  <ScrollViewer x:Name="ScrollViewer" Padding="{TemplateBinding Padding}" Background="{TemplateBinding Background}" BorderBrush="Transparent" BorderThickness="0" 
                                  TabNavigation="Cycle" IsTabStop="False">
                      <ItemsPresenter />
                  </ScrollViewer>
              </Border>
          </ControlTemplate>
      </ListBox.Template>
    <ListBox.ItemsPanel>
        <ItemsPanelTemplate>
            <VirtualCollection1:VirtualizingWrapPanel ItemWidth="200"
                                            ItemHeight="230"/>
        </ItemsPanelTemplate>
    </ListBox.ItemsPanel>
</ListBox>
The key part is right down at the bottom of the snippet, where we set the ItemsPanel property of the ListBox. There we specify an ItemsPanelTemplate which instantiates the VirtualizingWrapPanel. Notice that you need to tell the panel how big each item is going to be up front, so that it knows how much room to allow for items which are off screen.

There’s one other important point. Notice that I’ve overridden the Template property for the ListBox. This is so that I can tweak the ScrollViewer which is going to contain the VirtualizingWrapPanel. The tweak is a small one: I set its TabNavigation property to Cycle. Without this, keyboard navigation doesn’t work properly within the VirtualizingWrapPanel: under certain circumstances, navigating with the Up/Down or Page Up/Page Down keys would move the keyboard focus out of the ListBox, rather than to the next item in the ListBox.

Birds-Eye View of a Virtualizing Panel

So how do you implement a Virtualizing Panel in Silverlight?

There’s a post on my own blog which explains how to create Panels of the standard variety, and I suggest you go there first for a bit of background. Suffice to say here that normal Panels have one job in life: they must layout the elements given to them by their parent control.

Virtualizing Panels have two additional responsibilities. First, they must keep track of their own scroll state. And second, they must manage the creation of UI elements - item containers -for each data item as and when they are needed. They don’t actually create the elements – that is done by an ItemContainerGenerator. They just decide when each item container should be generated.

Implementing a virtualizing panel that can handle UI items of any size you throw at it is a very tricky job. That’s because, in order to accurately position any given element, you need to know the sizes of all the elements that come before it in the list. To know their sizes, you have to measure them, and to be measured, they need to exist. But the very reason for using a virtualizing panel is to avoid creating item unnecessarily. That is why virtualizing panels tend to take one of two easy routes out of the conundrum.

VirtualizingStackPanel’s favoured approach is to use index based scrolling, where it only ever shows whole items, and the scroll bar reflects the current items’ indexes in the list and has nothing to do with their sizes. This has the downside of jerky scrolling if you have large items, since each movement of the scrollbar will move previous items entirely out of view.

Jerky scrolling is avoided by the approach I’ve taken, which is to require that each item displayed by the panel is the same size. This enables us to easily calculate the position of any given item based on its index in the list.

Scrolling Responsibilities

How does a virtualizing panel discharge its scrolling responsibilities?

The panel needs to implement IScrollInfo so its containing ScrollViewer knows that it is going to manage its own scrolling. IScrollInfo’s job splits into two parts: one is to report on the current scroll state through properties like ExtentHeight and ViewportWidth. The Extent represents the logical size of the panel – what it would be if no scrolling was taking place and everything was visible at once. The Viewport is the rectangle of the panel that is actually visible at this moment.

IScrollInfo’s other job is to enable that scroll state to be manipulated, through methods like LineUp and MouseWheelDown. In many implementations, these manipulation methods all end up by calling one of two other methods on IScrollInfo, SetVerticalOffset and SetHorizontalOffset.

Both SetVerticalOffset and SetHorizontalOffset do much the same thing, so I’ll just show the SetVerticalOffset method:

public void SetVerticalOffset(double offset)
{
  if (_isInMeasure)
  {
      return;
  }

  offset = Clamp(offset, 0, ExtentHeight - ViewportHeight);
  _offset = new Point(_offset.X, offset);

  InvalidateScrollInfo();
  InvalidateMeasure();
}

After checking that the given offset is within the scrollable range, the method simply updates a field to show where the panel is scrolled to. Then it notifies its parent ScrollViewer that its state has changed, and, by calling InvalidateMeasure, tells the framework that its layout (i.e. the arrangement of its children) needs updating. The method does one other check: if it discovers that the panel is already in the middle of updating its layout (i.e. _isInMeasure is true) it doesn’t bother changing the scroll offset. This circumvents a problem I’ve encountered where Silverlight tries to scroll to elements newly created during the layout process.

Handling Layout

Layout happens in two parts. First we measure, calculating how much space the panel will occupy taking into account its children; then we arrange, allocating each child its place within the parent. Measuring happens in MeasureOverride, and arranging in ArrangeOverride. Most of the work in the VirtualizingWrapPanel actually happens in MeasureOverride because the panel needs to generate items containers before it can measure them. You might wonder why we need to measure the item containers when we’ve already decided that they have a fixed size? That is so that the item containers can handle their own layout, which happens using the same Measure and Arrange process.

Before I show you the code for MeasureOverride, here’s an outline of what it does:

  1. GetExtentInfo is called to calculate the full height of the list – i.e. how tall it would be if we displayed it all instead of scrolling.
  2. GetLayoutInfo then works out, based on the scroll offset, which item indexes fall inside the visible range. We always make sure to include realize one item before the first visible item, and one item after the last visible item. This enables keyboard scrolling to work properly, because Silverlight can then scroll that item into view, thus triggering the panel to realize the rest of the row.
  3. Any of the current item containers that now fall outside of the visible range are recycled – handed back to the ItemContainerGenerator so that it can reuse them for other items.
  4. MeasureOverride then loops through the visible item indexes and:
    1. Asks the ItemContainerGenerator to create the UI container for the item. If the visibility of the item hasn’t changed, ItemContainerGenerator will give us back the same item container we had before
    2. Calls SetVirtualItemIndex to tag the container with the index in the list that it is now representing. This is used in step 3 the next time round when we check for items that fall outside of the visible range.
    3. Makes sure the container is in the correct place in the Panel’s list of visual children, and thus part of the visual tree. If an item is not in the correct order, keyboard navigation won’t work correctly
    4. Asks the ItemContainerGenerator to Prepare the container – apply its styles, etc.
    5. Measures the item container – this must come after step 4, because until that point the item container isn’t fully initialized, and might not even know what its content is.
    6. Decide where exactly the item container is going to be displayed: its layout rectangle is recorded in the _childLayouts dictionary. This is used by the ArrangeOverride method to arrange each child
  5. Now we can go through the panel’s list of item containers and remove any which were never actually reused. Back in step 3 every existing item container’s VirtualItemIndex tag was set to –1; and in step 4.2, containers which are in use have their tag set to the item of the index they are now representing. So item containers which were never reused will still have a VirtualItemIndex of –1.
  6. Update the scroll state with the new extent height and viewport size so that scroll bars can be drawn in the correct proportions.
  7. Decide what size to report as the desired size for the Panel. The Panel wants to take up all the available space (within the confines of its parent), so in most cases, that’s what it reports. But if it is told that the available size is Infinite, then it returns a size of 0, and by convention, Silverlight takes that to mean “use all the available space”.

Here’s the code:

protected override Size MeasureOverride(Size availableSize)
{
  if (_itemsControl == null)
  {
      return availableSize;
  }
  
  _isInMeasure = true;
  _childLayouts.Clear();

  var extentInfo = GetExtentInfo(availableSize, ItemHeight);

  EnsureScrollOffsetIsWithinConstrains(extentInfo);

  var layoutInfo = GetLayoutInfo(availableSize, ItemHeight, extentInfo);

  RecycleItems(layoutInfo);

  // Determine where the first item is in relation to previously realized items
  var generatorStartPosition = _itemsGenerator.GeneratorPositionFromIndex(layoutInfo.FirstRealizedItemIndex);

  var visualIndex = 0;

  var currentX = layoutInfo.FirstRealizedItemLeft;
  var currentY = layoutInfo.FirstRealizedLineTop;

  using (_itemsGenerator.StartAt(generatorStartPosition, GeneratorDirection.Forward, true))
  {
      for (var itemIndex = layoutInfo.FirstRealizedItemIndex; itemIndex <= layoutInfo.LastRealizedItemIndex; itemIndex++, visualIndex++)
      {
          bool newlyRealized;

          var child = (UIElement)_itemsGenerator.GenerateNext(out newlyRealized);
          SetVirtualItemIndex(child, itemIndex);

          if (newlyRealized)
          {
              InsertInternalChild(visualIndex, child);
          }
          else
          {
              // check if item needs to be moved into a new position in the Children collection
              if (visualIndex < Children.Count)
              {
                  if (Children[visualIndex] != child)
                  {
                      var childCurrentIndex = Children.IndexOf(child);

                      if (childCurrentIndex >= 0)
                      {
                          RemoveInternalChildRange(childCurrentIndex, 1);
                      }

                      InsertInternalChild(visualIndex, child);
                  }
              }
              else
              {
                  // we know that the child can't already be in the children collection
                  // because we've been inserting children in correct visualIndex order,
                  // and this child has a visualIndex greater than the Children.Count
                  AddInternalChild(child);
              }
          }

          // only prepare the item once it has been added to the visual tree
          _itemsGenerator.PrepareItemContainer(child);

          child.Measure(new Size(ItemWidth, ItemHeight));

          _childLayouts.Add(child, new Rect(currentX, currentY, ItemWidth, ItemHeight));

          if (currentX + ItemWidth * 2 >= availableSize.Width)
          {
              // wrap to a new line
              currentY += ItemHeight;
              currentX = 0;
          }
          else
          {
              currentX += ItemWidth;
          }
      }
  }

  RemoveRedundantChildren();
  UpdateScrollInfo(availableSize, extentInfo);

  var desiredSize = new Size(double.IsInfinity(availableSize.Width) ? 0 : availableSize.Width,
                             double.IsInfinity(availableSize.Height) ? 0 : availableSize.Height);

  _isInMeasure = false;

  return desiredSize;
}

It’s a Wrap

That concludes our look at the VirtualizingWrapPanel. Go and check out the whole sample on GitHub, and let us know what you make of it.

Data Virtualization in Silverlight: Digging into VirtualCollection

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

The other day I showed you how we enabled the display of massive scrollable lists of items in Silverlight by implementing data virtualization for DataGrids and ListBoxes. If you’ve looked at that sample, you might be curious as to how it all works.

That is the subject for todays post.

What is Virtualization?

If you’re familiar with Silverlight or WPF, you’ll know all about virtualization when it comes to ListBoxes and DataGrids. Virtualization is a trick the framework uses to avoid having to render UI for items in a list that are not visible on screen. It creates UI for items that are visible, and as the user scrolls, it destroys the UI elements for data that has disappeared off screen, and creates new elements for items that have just come into view. VirtualizingStackPanel is one of the classes in WPF and Silverlight that is responsible for handling this on behalf of controls like the ListBox.

VirtualCollection, the class that lies at the heart of our solution, takes this one step further, and applies the same idea to the items in the list themselves. Rather than loading all the items into memory up front, it pulls in just the ones that are needed as the user scrolls (which might involve making a call to a server), and unloads any items which are not likely to be needed again any time soon. Sounds simple enough in theory, but getting this to work in Silverlight involves a bit of trickery.

Appetite reduction therapy, Silverlight style

The first problem we come up against is Silverlight’s I-want-it-and-I want-it-all-NOW attitude to data. If you present a Silverlight ListBox or DataGrid with a simple IList to be its data source, Silverlight will ask the list for its Count, and then proceed to request every single item in the list, irrespective of how much it can actually display at that moment. This isn’t a problem in most use cases, but when we want our VirtualCollection to display lists with potentially billions of data items, that approach just isn’t tenable. So we have to find a way to get Silverlight to slim down its appetite for data.

The trick here, it turns out, is to implement ICollectionView as well as IList. Flaunting this extra interface seems to be enough to persuade Silverlight to back down, and just ask for the items it can actually display, as it needs them. There is some extra work involved, as ICollectionViews are supposed to be responsible for keeping track of the current item selected in a list, but we can handle that.

So with ICollectionView implemented, Silverlight now asks the VirtualCollection for items it wants to display one by one. But that leads to the next problem. When a ListBox, say, first comes knocking on  the door,  VirtualCollection doesn’t have any items in stock. They’re all on the server, and have to be fetched into memory. But VirtualCollection can’t keep Silverlight waiting whilst it goes and does that, or Silverlight will sulk and hang the browser. So VirtualCollection fobs Silverlight off by giving it back a VirtualItem which acts as a place-holder for the real item. Silverlight can then happily create a row in the ListBox, albeit a blank row, because it has no actual data yet. When the real item arrives, fresh from the server, VirtualCollection hands it to the VirtualItem, and through the magic of data binding, the row lights up with the appropriate data.

Let’s take a look at some code, to see how VirtualCollection implements this process.

VirtualItems

We’ll start where Silverlight starts, when it wants to display a particular item: at VirtualCollection’s indexer:

public VirtualItem<T> this[int index]
{
  get
  {
      RealizeItemRequested(index);
      return _virtualItems[index] ?? (_virtualItems[index] = new VirtualItem<T>(this, index));
  }
  set { throw new NotImplementedException(); }
}

Two things are happening here: first up, we’re kicking off a request to realize the item – fetch it from the server, in other words. Secondly, we’re creating a VirtualItem to stand in for the real item – or if we’ve already created a VirtualItem representing that index, we simply return that.

It’s worth saying a word about how we store all these VirtualItems, because they are not stored in a standard List. The reason for that is that there could be billions of them, and we wouldn’t want to allocate an array that big (which is what a List would do). That would be especially wasteful given that the vast majority of the array would never be used, since the user is unlikely to look at every single item.

So VirtualItems are stored in a SparseList, which allows items to be accessed by index as conveniently as if they  were in one big array. Under the covers it actually stores items in lots of smaller arrays which can be more easily garbage-collected when they’re no longer needed.

To Fetch a Page of Data

Let’s chase the process along a little further.

public void RealizeItemRequested(int index)
{
  var page = index / _pageSize;
  BeginGetPage(page);
}

private void BeginGetPage(int page)
{
  if (IsPageAlreadyRequested(page))
  {
      return;
  }

  _mostRecentlyRequestedPages.Add(page);
  _requestedPages.Add(page);

  _pendingPageRequests.Push(new PageRequest(page, _state));

  ProcessPageRequests();
}

Items are never fetched from the server as individuals, but always in pages, so that we can make the most of each round-trip across the wire. The first thing that RealizeItemRequested does, therefore, is work out which page the requested index is part of. Then it puts in a request to fetch that page.

BeginGetPage doesn’t do any fetching directly. After ensuring it isn’t wasting time fetching a page it already has, what it does do is push a fetch-request for the page on to a stack, _pendingPageRequests. It also adds the page to a list it maintains of the mostly recently requested pages. This list is used to manage the cache of pages which are kept in memory. When the list gets too long, the oldest pages are evicted from memory.

Finally BeginGetPage goes off to process the page requests.

private void ProcessPageRequests()
{
  while (_inProcessPageRequests < MaxConcurrentPageRequests && _pendingPageRequests.Count > 0)
  {
      var request = _pendingPageRequests.Pop();

      // if we encounter a requested posted for an early collection state,
      // we can ignore it, and all that came before it
      if (_state != request.StateWhenRequested)
      {
          _pendingPageRequests.Clear();
          break;
      }

      // check that the page is still requested (the user might have scrolled, causing the 
      // page to be ejected from the cache
      if (!_requestedPages.Contains(request.Page))
      {
          break;
      }

      _inProcessPageRequests++;

      _source.GetPageAsync(request.Page * _pageSize, _pageSize, _sortDescriptions).ContinueWith(
          t =>
          {
              if (!t.IsFaulted)
              {
                  UpdatePage(request.Page, t.Result, request.StateWhenRequested);
              }
              else
              {
                  MarkPageAsError(request.Page, request.StateWhenRequested);
              }

              // fire off any further requests
              _inProcessPageRequests--;
              ProcessPageRequests();
          },
          _synchronizationContextScheduler);
  }
}

There are a number of interesting design decisions revealed by ProcessPageRequests, so it’s worth looking at in some detail.

First, why do we use a stack for managing the pending page requests, and not, say, a queue? Well, think about what happens when a user scrolls a long way through a list, and then stops. As they scroll, page requests will be generated, and will take some time to process, so the queue would grow. By the time the user stops scrolling, the queue might be quite long, and it would take a little time for the most recent requests, the ones for the pages the user is now staring at, to be dealt with. By putting the requests in a stack, which is a first-in-first-out data structure, we ensure that the most recent requests, those most likely to be relevant, are dealt with first.

We might also find that there are some requests which are no longer relevant. If the user scrolled a very long way, causing lots of pages to be added to the _mostRecentlyRequestedPages list, there may well be some pages which have to be kicked out of the cache. Any fetch requests for those pages can be safely ignored.

Requests can become irrelevant for another reason too: the collection state may have moved on since the request was made. The collection state is a concept that helps to make sure the VirtualCollection does the right thing at the right time in spite of all the asynchronous activity going on. The state is simply a number which is incremented every time VirtualCollection is notified by the IVirtualCollectionSource that the collection has changed (this is done in UpdateData and Reset). Whenever an asynchronous action is started, say a request to fetch data from the server, VirtualCollection tags the request with the state at that instant. When the response of the asynchronous action comes back, VirtualCollection compares its current state with the state when the action was begun. If the two are different, VirtualCollection knows that the world has moved on, so that response is no longer relevant.

Notice, finally, in this method that when we fire off the asynchronous request to get a page of data – this is where the IVirtualCollectionSource comes into play – we make sure the response is handled in the UI thread;  that is the purpose of passing _synchronizationContextScheduler to ContinueWith. All responses to asynchronous requests are handled in this way to make sure we don’t have any race conditions when updating VirtualCollection’s internal state.

Acting on the Data

What happens when the data is received from the server?

private void UpdatePage(int page, IList<T> results, uint stateWhenRequested)
{
  if (stateWhenRequested != _state)
  {
      // this request may contain out-of-date data, so ignore it
      return;
  }

  bool stillRelevant = _requestedPages.Remove(page);
  if (!stillRelevant)
  {
      return;
  }

  _fetchedPages.Add(page);

  var startIndex = page * _pageSize;

  for (int i = 0; i < results.Count; i++)
  {
      var index = startIndex + i;
      var virtualItem = _virtualItems[index] ?? (_virtualItems[index] = new VirtualItem<T>(this, index));
      if (virtualItem.Item == null || results[i] == null || !_equalityComparer.Equals(virtualItem.Item, results[i]))
      {
          virtualItem.SupplyValue(results[i]);
      }
  }

  if (results.Count > 0)
  {
      OnItemsRealized(new ItemsRealizedEventArgs(startIndex, results.Count));
  }
}

Data coming back from the server is passed to UpdatePage. Before doing anything with the data, the method checks that it is still relevant – that the collection state hasn’t changed, and that the page to which the data belongs hasn’t been evicted from the cache. Then it takes each item and passes it to its corresponding VirtualItem.

Two snippets from VirtualItem will suffice to show what’s going on there:

public class VirtualItem<T> : INotifyPropertyChanged where T : class
{
    // Snip ...
    
   public T Item
   {
       get
       {
           if (!IsRealized && !DataFetchError)
           {
               _parent.RealizeItemRequested(Index);
           }
           return _item;
       }
       private set
       {
           _item = value;
           OnPropertyChanged(new PropertyChangedEventArgs("Item"));
           OnPropertyChanged(new PropertyChangedEventArgs("IsRealized"));
           IsStale = false;
       }
   }

   public void SupplyValue(T value)
   {
       DataFetchError = false;
       Item = value;
   }
}

VirtualItem makes the real item available through its Item property, and when SupplyValue is called this leads to a PropertyChanged event being raised for the Item property, so the Silverlight databinding infrastructure knows it needs to update the UI for that item.

Handling Updates

There’s one other important aspect of VirtualCollection to consider, and that is how it handles changes to the underlying collection of data.

It is the responsibility of the IVirtualCollectionSource to tell VirtualCollection when these changes happen.

The events are handled here:

private void HandleSourceCollectionChanged(object sender, VirtualCollectionSourceChangedEventArgs e)
{
  var stateWhenUpdateRequested = _state;
  if (e.ChangeType == ChangeType.Refresh)
  {
      Task.Factory.StartNew(() => UpdateData(stateWhenUpdateRequested), CancellationToken.None,
                            TaskCreationOptions.None, _synchronizationContextScheduler);
  }
  else if (e.ChangeType == ChangeType.Reset)
  {
      Task.Factory.StartNew(() => Reset(stateWhenUpdateRequested), CancellationToken.None,
                            TaskCreationOptions.None, _synchronizationContextScheduler);
  }
}

There are two types of changes we consider, a Refresh, and a complete Reset. Refresh happens when much of the data may well have remained unchanged, and it makes sense to continue to display it whilst we check with the server. A Reset happens when the entire collection has changed and displaying stale items would be inappropriate.

Notice that we are dispatching the events to be handled on the UI thread (there’s that _synchronizationContextScheduler again), and as we do so, we tag the request with the current collection state so that when we get round to handling the requests we can ignore any that are no longer relevant.

The Reset method contains nothing of great interest, but UpdateData explains something quite important:

protected void UpdateData(uint stateWhenUpdateRequested)
{
  if (_state != stateWhenUpdateRequested)
  {
      return;
  }

  _state++;

  MarkExistingItemsAsStale();

  _fetchedPages.Clear();
  _requestedPages.Clear();

  UpdateCount();

  var queryItemVisibilityArgs = new QueryItemVisibilityEventArgs();
  OnQueryItemVisibility(queryItemVisibilityArgs);

  if (queryItemVisibilityArgs.FirstVisibleIndex.HasValue)
  {
      var firstVisiblePage = queryItemVisibilityArgs.FirstVisibleIndex.Value / _pageSize;
      var lastVisiblePage = queryItemVisibilityArgs.LastVisibleIndex.Value / _pageSize;

      int numberOfVisiblePages = lastVisiblePage - firstVisiblePage + 1;
      EnsurePageCacheSize(numberOfVisiblePages);

      for (int i = firstVisiblePage; i <= lastVisiblePage; i++)
      {
          BeginGetPage(i);
      }
  }
  else
  {
      // in this case we have no way of knowing which items are currently visible,
      // so we signal a collection reset, and wait to see which pages are requested by the UI
      OnCollectionChanged(new NotifyCollectionChangedEventArgs(NotifyCollectionChangedAction.Reset));
  }
}

Remember that the purpose of UpdateData is to check that each item in the collection remains up to data. But it has to be frugal: it can’t go off to the server to ask about every possible item. Once again, it needs to limit itself to just those items that are visible.

One way of doing this, the way VirtualCollection worked when I first implemented it, is to raise a CollectionChanged event with an NotifyCollectionChangedAction of Reset. This signals to any listening ListBoxes or DataGrids that the whole collection has changed and they should rebuild their display of the data. They then respond by calling VirtualCollection’s indexer for each index  in the visible range, and thus VirtualCollection finds out which pages it should refresh. In many circumstances this is the only way of refreshing the collection because we have no other way of knowing exactly which items are visible.

But it is a rather sledgehammer approach, rebuilding all the rows in the UI, when we could just update the data within those rows, if only we knew which items are currently visible. So before resorting to a collection reset, VirtualCollection will try another approach. It tries asking nicely anybody who might be listening which items are visible.

It does this by raising the QueryItemVisibility event. To make sure this request doesn’t go unheeded, you need to attach a behaviour to your DataGrid or ListBox which will respond to the event. As part of the code sample, I have provided the ProvideVisibleItemRangeFromDataGridBehavior and the ProvideVisibleItemRangeFromItemsControlBehavior – though note that currently, this last behavior only does anything useful if the ItemsControl (usually in the form of a ListBox) is using a VirtualizingWrapPanel to display its items. See MainPage.xaml in the sample for an example of how to attach these behaviors.

The Final Word

And that concludes our dissection of VirtualCollection. Remember that you can try out a live demo here, and get all the code on github so that you can put it to use in your own projects.

Do let us know how you get on.

Next time: exploring the VirtualizingWrapPanel.

Data Virtualization, Lazy Loading, Stealth Paging–whatever you want to call it: Here’s how to do it in Silverlight

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

As I have previously mentioned, the management UI for RavenDb (fondly known as the Studio) has a hot new feature. A revolutionary interface for viewing huge sets of documents which uses (gasp!) scrollbars instead of paging controls. To make this feature kind on the server (we’re averse to killing Ravens of any kind), the UI requests documents just in time, as the users scrolls each part of the list into view.

In the next few blog posts, I want to share with you how we made that work in Silverlight. I’ve published a sample on GitHub, including code under a license that enables you to use it in your own projects.

Today, to whet your appetites, I’ll show you how easy it is to create user interfaces that display huge collections of items. In future posts, I’ll dig into the magic which makes it all work.

Flicking through the Netflix Catalogue

The sample is a simple browser for the Netflix movie catalogue, which lets you scroll through all the available titles, or just those containing the search terms you enter. I chose this example because Netflix have an OData API which makes the catalogue very easy to query.

As well as the details view, shown below, the sample also has a card view which demonstrates our VirtualizingWrapPanel, also included in the sample.

Netflix Browser Sample

Click the image to see the sample in action. (There’s a known issue with images not showing up, because of how I’m hosting the files in Amazon S3 – run the sample locally to see it in its full glory)

Holding it all together: VirtualCollection

The key player in this setup is the VirtualCollection class, which you’ll find in the sample code. VirtualCollection acts as the data source which you bind to your ListBoxes or DataGrids, and it has the responsibility of coordinating data retrieval from the server.  The actual fetching of data is handled by a class implementing IVirtualCollectionSource: each VirtualCollection is associated with one IVirtualCollectionSource.

public class MainViewModel : ViewModel
{
   private NetflixTitlesSource _source;
   public VirtualCollection<Title> Items { get; private set; }

   public MainViewModel()
   {
       _source = new NetflixTitlesSource();
       Items = new VirtualCollection<Title>(_source, pageSize: 20, cachedPages: 5);
   }

   protected override void OnViewLoaded()
   {
       Items.Refresh();
   }
}

Using the VirtualCollection is very straightforward. You create an instance, telling it what kind of items it will be managing (Title objects in this case), and supplying it with the IVirtualCollectionSource (implemented here by NetflixTitlesSource – more on this in a moment).

You also tell it what page size to use, in other words, how many items it should fetch in each call to the server. You can determine an appropriate page size by figuring out how many items are likely to fill one screen of your application. VirtualCollection also wants to know how many such pages it should cache in memory: if your items are small, and change infrequently, caching more pages might well be a good idea.

Supplying the goods: IVirtualCollectionSource

When it comes to implementing IVirtualCollectionSource, you have to do a little more work, but nothing too arduous. In fact, there’s a base class which does much of the housekeeping for you.

An IVirtualCollectionSource has two main responsibilities: it needs to inform the VirtualCollection of the total number of items, and it must supply pages of items when requested. It should also let the VirtualCollection know if anything changes, by means of the CollectionChanged event.

Here’s what the key methods of the NetflixTitlesSource look like:

public class NetflixTitlesSource : VirtualCollectionSource<Title>
{
    private string _search;

   public string Search
   {
       get { return _search; }
       set
       {
           _search = value;
           Refresh(RefreshMode.ClearStaleData);
       }
   }
   
   protected override Task<int> GetCount()
   {
       return GetQueryResults(0, 1, null)
           .ContinueWith(t => (int)t.Result.TotalCount, TaskContinuationOptions.ExecuteSynchronously);
   }

   protected override Task<IList<Title>> GetPageAsyncOverride(int start, int pageSize, IList<SortDescription> sortDescriptions)
   {
       return GetQueryResults(start, pageSize, sortDescriptions)
           .ContinueWith(t => (IList<Title>)((IEnumerable<Title>)t.Result).ToList(), TaskContinuationOptions.ExecuteSynchronously);
   }

    private Task<QueryOperationResponse<Title>> GetQueryResults(int start, int pageSize, IList<SortDescription> sortDescriptions)
   {
       var context = new NetflixCatalog(new Uri("http://odata.netflix.com/Catalog"));

       var orderByString = CreateOrderByString(sortDescriptions);
       var query = context.Titles
           .AddQueryOption("$skip", start)
           .AddQueryOption("$top", pageSize)
           .IncludeTotalCount();

       if (!string.IsNullOrEmpty(Search))
       {
           query = query.AddQueryOption("$filter", "(substringof('" + Search + "',Name) eq true) and (BoxArt/SmallUrl ne null)");
       }
       else
       {
           query = query.AddQueryOption("$filter", "(BoxArt/SmallUrl ne null)");
       }

       if (orderByString.Length > 0)
       {
           query = query.AddQueryOption("$orderby", orderByString);
       }

       return Task.Factory.FromAsync<IEnumerable<Title>>(query.BeginExecute, query.EndExecute, null)
           .ContinueWith(t => (QueryOperationResponse<Title>)t.Result, TaskContinuationOptions.ExecuteSynchronously);
   }
   
   ...
}

As you can see, the majority of the work work lies in preparing the query that we send off to the Netflix OData API, and executing that asynchronously . By comparison, the code needed to support IVirtualCollectionSource (just two methods, GetCount and GetPageAsyncOverride) is very small.

A VirtualCollectionSource can communicate with its parent VirtualCollection by means of the CollectionChanged event if something happens that means the collection needs to be updated. In this case, changing the Search property is going to change the entire collection, so we call the Refresh method on the base class, which raises that event.

There are two kinds of Refresh available. The Search property is using the ClearStaleData mode, which means that the whole collection will be cleared whilst new results are being loaded. This is obviously the right choice here, because continuing to show results for “Harry Potter” after the user has just changed the search terms to “Indiana Jones” would be confusing.

The other kind of Refresh is called PermitStateDataWhilstRefreshing. This mode is good if your server has just notified you that new items have been added. Existing items are still valid, so there’s no need to clear down the whole collection whilst you find out about the new items.

Making it Presentable

Finally, here’s some XAML to show how we might display the contents of the VirtualCollection in a DataGrid

<sdk:DataGrid ItemsSource="{Binding Items}" Grid.Row="1" Margin="10,0" IsReadOnly="True" AutoGenerateColumns="False" RowHeight="50">
 <i:Interaction.Behaviors>
     <VirtualCollection1:ProvideVisibleItemRangeFromDataGridBehavior/>
 </i:Interaction.Behaviors>
 <sdk:DataGrid.Columns>
     <sdk:DataGridTemplateColumn Header="Image">
         <sdk:DataGridTemplateColumn.CellTemplate>
             <DataTemplate>
                 <Image Stretch="None" 
                         Source="{Binding Item.BoxArt.SmallUrl, Converter={StaticResource Converter_StringToImage}}" 
                        Margin="5" />
             </DataTemplate>
         </sdk:DataGridTemplateColumn.CellTemplate>
     </sdk:DataGridTemplateColumn>
     <sdk:DataGridTextColumn Binding="{Binding Item.Name}" Header="Name" CanUserSort="True" SortMemberPath="Name"/>
     <sdk:DataGridTextColumn Binding="{Binding Item.AverageRating}" Header="Average Rating" CanUserSort="True" SortMemberPath="AverageRating"/>
     <sdk:DataGridTextColumn Binding="{Binding Item.ReleaseYear}" Header="Release Year" CanUserSort="True" SortMemberPath="ReleaseYear"/>
 </sdk:DataGrid.Columns>
</sdk:DataGrid>

The main thing to notice here is that all the binding paths start with Item. That’s because each data item in the VirtualCollection is wrapped in a VirtualItem, and made available through VirtualItem’s Item property. VirtualCollection does this so that it can put off fetching any actual items until the last possible moment – more on this in another blog post.

Notice also that we can enable sorting on some of the columns by setting a SortMemberPath. When you click a column header, the DataGrid will pass that value to the VirtualCollection, and VirtualCollection will pass it on to the IVirtualCollectionSource which can then use it to instruct the server in how to sort the results it returns.

Try it out and Get Back to Us

Now you’ve seen how easy it is, you have no excuse. Go and purge those paging controls from your apps! Long live scrollbars!

All the code you need is up on GitHub - look in the VirtualCollections folder. And don’t forget to check back here shortly, as we take a look behind the scenes to find out how VirtualCollection works.

We’d love to hear how you get on with this in your own projects.

New in RavenDb Studio 1.2: Low-Bandwidth mode

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

We saw earlier that in RavenDb Studio 1.2 you can now scroll through the entire list of documents in a Raven database, and the Studio will page in documents just in time behind the scenes. But those documents still need to be fetched from the server, and if your documents are very large (into the 100s of kilobytes and more), or you are working over a very slow connection, fetching the documents will take some time, and make for a frustrating experience.

Which is why we’ve introduced a low-bandwidth, or “Id only” mode for viewing document lists. You switch to it by clicking the Document View Style button on the top right of any Document list.

image

In this mode, the Studio will only fetch the metadata for each document, saving a lot of time when contacting the server and allowing the lists to be displayed much more quickly. To see the whole document, you simple double-click it, or click the Pencil icon, as in Details or Card mode.

Published at

Originally posted at

New in RavenDb Studio 1.2: Quick Links

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

Here’s another small, but perfectly formed feature in RavenDb Studio 1.2.

The header bar now sports a “quick links” area:

image

Two things to see here. First there’s the new New button which speeds you on your way to create a new document, index or query.

image

And there’s the “Go To Document” bar. Type the Id of a document, hit Enter, and you’ll be taken straight to it. Even better, if you don’t know the exact name of the document, start typing its Id, and you’ll be prompted with a list of possibilities.

image

Published at

Originally posted at

New in RavenDb Studio 1.2: Paging through result sets

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

A small, but very useful enhancement today, in our on-going tour through the new UI features for RavenDb 1.2

image

Notice the paging controls highlighted in the screenshot? These now appear whenever you arrive at a document via a documents list. This might be from the Documents or Collections page, for example, or from the results list of a Query. These controls let you page through all the documents in that particular list.

Say, for example, that you have misspelt an Artist’s name and you need to make a quick correction in the database. You could create a query to find all the documents containing the misspelling, and then double-click the results list to edit the first one. You make the correction, and save the document. What do you do next? Before, you would have had to hit the back button to go back to the query page to find the next document. Now you can just hit the Next button in the Edit page to work through the erroneous documents one-by-one.

New in RavenDb Studio 1.2: Better document editing

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

Continuing in our tour of the new features in RavenDb Studio for 1.2, today we come to the document editor.

Outlining Support

Since Documents are RavenDb’s raison d'etre, having a good document editing and viewing experience is vital. Which is why we’ve put a good bit of work into enhancing the Studio’s document editor.

The first thing you’ll notice is that the editor now has outlining support, so that you can expand and collapse collections and nested objects within a document:image_thumb35

Clicking the Outlining button collapses all the top level nodes, and clicking it again expands them all.

If you have many documents containing large collections, you may find the new Auto-Collapse Collections feature useful. When this mode is selected (find it by clicking the drop-down arrow on the Outlining button) the Document editor will automatically collapse all collections when you first open a document in the editor page.

JSON Syntax Error Highlighting

The next new feature you probably won’t notice, until you start modifying a document. Then you’ll see that the editor now has real-time JSON syntax checking:

image_thumb37

Any errors in the document are highlighted with red squiggles, and an error list at the bottom summaries all the errors detected, with double-click taking you to the mal-formed part of the document, as you’d expect.

Document Hyperlinks

Here’s a handy new feature for checking relationships between documents. The document editor now turns any document Ids it encounters within a document into hyperlinks that you can navigate to with a Shift-Click (or Shift-Ctrl-Click to open in a new page):

image

 

Not only is this good for jumping between related documents. It also lets you see at glance if your document references are valid, since the document editor only hyperlinks Ids of documents that actually exist in the database.

Published at

Originally posted at

New in RavenDb Studio 1.2: Enhancements to querying

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

Welcome back to our tour of the new features in RavenDb Studio 1.2. Up for examination today is the Query page.

Remembering Recent Queries

Over on the RavenDb issue tracker, we were asked to “treat queries as first-class citizens”. I don’t know what other rights queries might demand, but one right we decided to give them was the right to be remembered. Enter the Recent Queries dropdown:

image

Every query you execute is remembered, and you can go back to a previous query just by clicking its entry in the list. As with most navigation buttons in the Studio, clicking with the Control key held down will open the query in a new window.

Up to 100 of your most recent queries are remembered for next time after you close your browser. Though if you should have been querying with some nefarious purpose, there’s always the “Clear History” button to cover your tracks.

If you have any queries that are especially near and dear to your heart, you can Pin them to make sure they always stay at the top of the list and don’t get swept away if the rest of the history is cleared.

 

Other Changes

Along with this, there are a number of smaller enhancements

  • Sort By now works for dynamic queries when querying over a particular collection.
  • Similarly, you now get Intellisense in the query editor for dynamic queries when you have limited the query to a specific collection. In other words, press Ctrl-Space in the query editor, and you can choose from a list of properties in documents in that collection.
  • Sort By options are now remembered when you navigate back to a previous query.
  • You can now choose the default operator used to combine clauses in your query .
  • A Show Fields option has been added to help you understand what’s going on with your indexes. When pressed, the results view will show the fields in matching index entries, rather than the actual documents. In a similar vein, there’s a new Skip Transform button which shows up when an Index has a Transform stage defined. Pressing Skip Transform will show you the documents matching a query, rather than the results of the transform.
  • For all you JSON-heads out there, the Query page shows you what url the client is calling to get the results from the server, and if you press the link button next to the url, you’ll see the raw JSON that the server sends back.

New in RavenDb Studio 1.2: Scrollable Document Lists, and a Details View

This is a guest post by Samuel Jack, a freelancer who has worked with us on the new RavenDb Studio features.

Along with the exciting new features in the core RavenDb database, we’ve been hard at work improving RavenDb Studio, the management UI for RavenDb. Over the next few days, I want to share with you some of the major enhancements.

Death to paging!

Almost the very first thing we did when we started work on the Studio for 1.2 was to kill the paging interface for document lists. And hammer nails in its coffin. Please give a big round of applause as we introduce … scrollbars!

image_thumb3

That’s right: scrollbars that let you scroll through your entire collection of documents, all 2,147,483,647 of them. Don’t be alarmed, you db admin types: the Studio is doing the right thing and paging in documents only as you scroll to them (and paging out documents you’ve scrolled past, so they don’t clog up memory).

 

Digging into the Document Details

By popular request, we’ve also added a details view as an alternative to the existing card view. Click the Document View Style button in the top right corner of any documents list and you can toggle between card view and details view (which ever setting you choose is remembered between sessions).

image_thumb12

Which columns get shown in details view? Well, naturally the Studio allows you to choose for yourself which columns you want to see. Or you can let the Studio choose for you.

 

 

Pick your own Columns

To choose your own columns, right click on one of the column headers and click the Choose Columns menu item. You’ll see a dialog where you can choose the columns you’d like:

image_thumb17

When you click in the Binding column to add a new Column, the Studio will help you out by showing a list of all the properties (and nested properties) in the visible documents:

image_thumb19

Notice that you can bind to document meta-data too!

 

Automatic Column Selection

To save you having to pick columns yourself for each set of documents you view, the Studio has a special IntelliPickNot TM feature. It will inspect the properties of documents currently scrolled into view, and do its best to pick a sensible set of columns. And as you scroll, and different types of documents come into view, it will automatically update the column set.

How does it choose columns? Good question. Let me check the documentation source code…

Basically it picks the top 6 most commonly occurring properties in the current set of documents. But it will give certain properties a boost in importance making them more likely to be picked. The Studio has a built in list of property names that it will look for. So any property that includes the words “Name”, “Title” “Description”, or “Status” will be prioritised. You can customise this list of priority columns for a database  by creating a document  in that database called “Raven/Studio/PriorityColumns”, like this:

image_thumb22

PropertyNamePattern is interpreted as a Regular Expression so you can get quite sophisticated in properties you want to boost.

The IntelliPick algorithm has one more trick up its sleeve. Have a look at this screenshot:

image_thumb30

Do you see it? If you include particular properties in a query, or as a Sort By option, those properties will get boosted in importance. In the example shown, Artist.Name is part of the query, and results are sorted by Genre.Name, so both these properties are picked out to appear as columns. Similarly, when viewing results for a particular index, any properties that are included in the Index name will also be given a boost in priority.

Fixing memory leaks in RavenDB Management Studio - FluidMoveBehavior

Continuing on my last blog post in this series, which I talked about the WeakReference, that time I’ll talk about the FluidMoveBehavior.

The FluidMoveBehavior gives you a great transition effect to the items in your WrapPanel, which is in the Silverlight toolkit. The FluidMoveBehavior is part of the Expression Blend and it’s exists in the microsoft.expression.interactions.dll.

When I profiled the application with a memory profiler, I have some memory leaks that caused by the FluidMoveBehavior. Surprised I Googled the following “FluidMoveBehavior memory leak” and the first result was this thread, which effectively showed that this is a known issue with no fix yet.

So removing the FluidMoveBehavior from the Management Studio fixes a big source of memory leak. What’s interesting, that the visual effect itself of the FluidMoveBehaviour barley was needed, since we already populating the panel with items each time the panel size is changed.

Published at

Originally posted at

Fixing memory leaks in RavenDB Management Studio - WeakReference

Continue from the last blog post in this series, which I talked about the WeakEventListener, now I’m going to talk about using the WeakReference.

In the RavenDB Management Studio we have 4 pages that contains lots of data: the Home page, Collections page, Documents page and Indexes page. Once you enter to one of those pages, we’ll fetch the data from the RavenDB database but in order to avoid fetching it each time we navigate to that page the data is stored in a static variable. This way, if you re-navigate to a page, you will see the database immediately while we making a background request to RavenDB in order to give you more updated data.

You can look on this code for example:

public class HomeModel : ViewModel
{
    public static Observable<DocumentsModel> RecentDocuments { get; private set; }

    static HomeModel()
    {
        RecentDocuments = new Observable<DocumentsModel>
                          {
                            Value = new DocumentsModel
                                    {
                                        Header = "Recent Documents",
                                        Pager = {PageSize = 15},
                                    }
                          };
    }

    public HomeModel()
    {
        ModelUrl = "/home";
        RecentDocuments.Value.Pager.SetTotalResults(new Observable<long?>(ApplicationModel.Database.Value.Statistics, v => ((DatabaseStatistics)v).CountOfDocuments));
        ShowCreateSampleData = new Observable<bool>(RecentDocuments.Value.Pager.TotalResults, ShouldShowCreateSampleData);
    }

    public override Task TimerTickedAsync()
    {
        return RecentDocuments.Value.TimerTickedAsync();
    }
}

The problem is, what happening when the application consumes to much memory because of all of this static data? In that case there likely to be a performance issues. In order to avoid that, we used make static data to be a WeakReference type, so we basically say to the Silverlight GC engine: If you want to GC the data, please do so. And in this case we’ll just re-initialize it when we need the data again.

This had an huge impact of the memory consumption of the Management Studio application, but we still had some memory leaks which I’ll talk about in the next blog post.

Published at

Originally posted at

Fixing memory leaks in RavenDB Management Studio - WeakEventListener

After shipping the new version of the Management Studio for RavenDB, which was part of build #573, we got reports from our users that it have some memory leaks. This report indicated that we have an huge memory leak in the management studio. I started to investigate this and found a bunch problems with cause it. In this blog posts series I’ll share with you what it took to fix it.

RavenDB Management Studio is a Silverlight based application. One of the mistakes that can be done easily in a Silverlight application (as many other platforms for UI applications) is to attach an event to an object, than discard that object. The problem is that the object will never be cleaned up from the memory, since we have a reference for it – the event listener.

Consider the following code for example:

public static class ModelAttacher
{
    public static readonly DependencyProperty AttachObservableModelProperty =
        DependencyProperty.RegisterAttached("AttachObservableModel", typeof(string), typeof(ModelAttacher), new PropertyMetadata(null, AttachObservableModelCallback));
    
    private static void AttachObservableModelCallback(DependencyObject source, DependencyPropertyChangedEventArgs args)
    {
        var typeName = args.NewValue as string;
        var view = source as FrameworkElement;
        if (typeName == null || view == null)
            return;

        var modelType = Type.GetType("Raven.Studio.Models." + typeName) ?? Type.GetType(typeName);
        if (modelType == null)
            return;

        try
        {
            var modelInstance = Activator.CreateInstance(modelType);
            var observableType = typeof(Observable<>).MakeGenericType(modelType);
            var observable = Activator.CreateInstance(observableType) as IObservable;
            var piValue = observableType.GetProperty("Value");
            piValue.SetValue(observable, modelInstance, null);
            view.DataContext = observable;

            var model = modelInstance as Model;
            if (model == null) 
                return;
            model.ForceTimerTicked();

            SetPageTitle(modelType, modelInstance, view);
            
            view.Loaded += ViewOnLoaded;
        }
        catch (Exception ex)
        {
            throw new InvalidOperationException(string.Format("Cannot create instance of model type: {0}", modelType), ex);
        }
    }
    
    private static void ViewOnLoaded(object sender, RoutedEventArgs routedEventArgs)
    {
        var view = (FrameworkElement)sender;
        var observable = view.DataContext as IObservable;
        if (observable == null)
            return;
        var model = (Model)observable.Value;
        model.ForceTimerTicked();

        var viewModel = model as ViewModel;
        if (viewModel == null) return;
        viewModel.LoadModel(UrlUtil.Url);
    }

For information about the ModelAttacher pattern, take a look on this blog post.

What it means basically is that we creating a models for each page, but never dispose it. So basically each time you navigate to a page, a new view model is created for the page but the old one never got cleaned up.

There was more examples like that, where we have an event keeping a reference to a dead objects. You can look on the RavenDB commits log if you interested in the details. But what is the way to solve this?

In order to solve this I copy the WeakEventListener from the Silverlight Toolkit, which is internal class. Using the WeakEventListener in order to attach to objects solved the above memory issue since we don’t have a strong reference to the dead object anymore, and the GC can just clean them up.

Model Attacher pattern in Silverlight applications

A while ago, when we started to develop our next version of RavenDB Studio, one of our goals was to make its code as simple as possible. That way, we ensure that it is easy to understand what is going on, so making changes to the Studio should be a trivial task.

In order to achieve that, we decided to not use any MVVM toolkits, but simply use a simple pages (views) and attach a model to them. In this approach, every view (page) know how to resolve its view-model by itself. This makes the Silverlight code much more simple, since it let’s you open a specific view by just navigating to its relative URL.

In order to make this possible we have a ModelAttacher.AttachModel attached property on the page, which takes care of instantiating the view-model and attach it to the page’s DataContext property.

Take a look on the following tipical view (page) in order to see it in action:

<Infrastructure:View x:Class="Raven.Studio.Views.Home"
                     xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
                     xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
                     xmlns:Infrastructure="clr-namespace:Raven.Studio.Infrastructure"
                     Title="Home"
                     Style="{StaticResource PageStyle}"
                     Infrastructure:ModelAttacher.AttachModel="HomeModel">

</Infrastructure:View>

In this example, we have an empty Home view, which is make a use of a HomeModel. The ModelAttacher’s job here is to create an instance of the HomeModel class and attach it to the View.DataContext property. (The view is a simple class that derives from Page.)

This is how ModelAttacher works, in order to achieve this:

namespace Raven.Studio.Infrastructure
{
    public static class ModelAttacher
    {
        public static readonly DependencyProperty AttachModelProperty =
            DependencyProperty.RegisterAttached("AttachModel", typeof(string), typeof(ModelAttacher), new PropertyMetadata(null, AttachModelCallback));
        
        private static void AttachModelCallback(DependencyObject source, DependencyPropertyChangedEventArgs args)
        {
            var typeName = args.NewValue as string;
            var view = source as FrameworkElement;
            if (typeName == null || view == null)
                return;

            var modelType = Type.GetType("Raven.Studio.Models." + typeName) ?? Type.GetType(typeName);
            if (modelType == null)
                return;

            try
            {
                var model = Activator.CreateInstance(modelType);
                view.DataContext = model;
            }
            catch (Exception ex)
            {
                throw new InvalidOperationException(string.Format("Cannot create instance of model type: {0}", modelType), ex);
            }
        }

        public static string GetAttachModel(UIElement element)
        {
            return (string)element.GetValue(AttachModelProperty);
        }

        public static void SetAttachModel(UIElement element, string value)
        {
            element.SetValue(AttachModelProperty, value);
        }
    }
}

Now in order to attach a model to its view, we need to just add the attached property to the view element:  Infrastructure:ModelAttacher.AttachModel="HomeModel".

Please note that in this case any view-model has to have a default (parameters-less) constructor. In order to solve that, what we have done in RavenDB Studio is to have a ModelBase class for our view-models which make sure to expose all our common model dependencies.