shared_ptr and aliasing constructor

shared_ptr seems to be straightforward class, but if you look closer you will find aliasing constructor that make you think – what the hell is that?

template<class Y>
shared_ptr( const shared_ptr<Y>& r, T *ptr );

The idea of this constructor is to keep reference count of owned object – r, while using pointer of some other stored object (mostly field in r object). This means that shared_ptr is allowed to own one object and store some other raw pointer. Only owned object is reference counted, while stored pointer is the one that gets returned during dereferencing. In normal situation, when aliasing constructor is not used, raw pointer is the one for owned object.

Below is example of aliasing constructor usage, where stored pointer is contained inside owned one:

#include <iostream>
#include <memory>

struct Data {
	int d;
};

int main(int argc, char* argv[])
{
	std::shared_ptr<Data> pd(new Data());
	std::shared_ptr<int> pd_d(pd, &pd->d); //pd is owned object, while &pd->d is stored (retuned on dereference)

	*pd_d = 10; 

	std::cout << pd.use_count() << std::endl; // outputs 2
	pd_d.reset();
	std::cout << pd.use_count() << std::endl; // outputs 1
	pd.reset();
	std::cout << pd.use_count() << std::endl; // outputs 0

	// pd->d is now a dangling pointer, dont use it!

	return 0;
}

Other use is for making shared_ptr of derived type:

        struct Base {
	  virtual ~Base(){}
        };
        struct Derived : public Base {
	  void foo() {}
        };

	std::shared_ptr<Base> pd_base(new Derived());
	std::shared_ptr<Derived> pd_derived(pd_base, dynamic_cast<Derived*>(pd_base.get()));
	pd_derived->foo();

Other use case for aliasing constructor is when you are using interface that requires shared_ptr but you dont want it to keep ownership of your object. In such case you can make shared_ptr to work like normal bare pointer:

    static Data globalData;
    std::shared_ptr pd2(std::shared_ptr(), &globalData.d);
    std::cout << pd2.use_count() << std::endl; // outputs 0, but *pd is valid
    *pd2 = 2;

Aliasing constructor is not forcing you to keep pointer as a stored object, it can be any type of size_t. When you realize this you can use this constructor also for keeping timestamp in some caching mechanism. For further info look here:

http://www.codesynthesis.com/~boris/blog/2012/04/25/shared-ptr-aliasing-constructor/

Other uses are: refering to container element (owned object is container, stored pointer is element in container), referring to a shared library symbol.

What you should not do:

	std::shared_ptr pd(new Data());
	std::shared_ptr pd_d(pd, new int()); // here new int() is an error, this pointer will never be freed.

Advertisement

gitstats filter files

Currently gitstats does not allow to filter files by extensions, and the only way is as author suggests:

https://github.com/dcramer/gitstats/blob/master/doc/gitstats.pod
Q: I have files in my git repository that I would like to exclude from the statistics, how do I do that?
A: At the moment the only way is to use git-filter-branch(1) to create a temporary repository and generate the statistics from that.

below is a short script on how to prepeare your repository for such filtered gitstats statistics calcualtion:

i assume you have copied your repository to some temporary folder, actually all below changes can be reverted but to be safe do it in temporary dir.

# clear file list to be removed from statistics calculation
$ rm ~/temp_gitstats_filter.txt

# get latest changes
$ git pull

# create a list of files excluding the ones ending with java and xml, those files will not be processed by gitstats
$ for i in `git ls-files | grep -Ev ‘^.*.(java|xml)$’`; do echo “$i” >> ~/temp_gitstats_filter.txt; done

# remove all the files from previously created list in current repository
$ git filter-branch -f –tree-filter ‘cat ~/temp_gitstats_filter.txt | xargs rm -f’ HEAD

# now execute your gitstats command (should be fixed for your needs)
$ gitstats .git /var/www/html/git_stats

# reset all the changes filter-branch have done
$ git reset –hard refs/original/refs/heads/master

some usefull links:
http://ktln2.org/blog/post/remove-files-from-history-with-filter-branch/
http://git.661346.n2.nabble.com/Remove-all-files-except-a-few-files-using-filter-branch-td7567155.html
http://stackoverflow.com/questions/4228100/git-i-want-to-unstage-all-files-matching-a-certain-pattern
http://unix.stackexchange.com/questions/37313/how-do-i-grep-for-multiple-patterns

rvalue, lvalue, prvalue, xvalue, glvalue – what are those?

Previous standard C++03 divided expressions into lvalues and rvalues, new standard C++11 adds three more categories: xvalue, glvalue and prvalue. What are those and why we need them all?

Lets start with lvalues and rvalues.

The common definition for lvalue is a non-const object that may be put on the left hand side of assignment. It can be an expression or function.

rvalue on the other hand is an expression that cannot be placed on the left hand side of assignment, it provides only value and cannot be assigned to.

For example:

int lvalue;
lvalue = 1;         // 1*
(lvalue + 10) = 1;  // 2* - error

char arr[15];
*(arr + 3) = 'a' ; // 3*

int& lvalueRetFunc(int& v) { return v; }
int rvalueRetFunc(int v) { return v; }
lvalueRetFunc(lvalue) = 10; // 4*
rvalueRetFunc(lvalue) = 10; // 5* - error

1* – here we have proper assignment to lvalue
2* – here we have an expression (lvalue + 10) that denotes rvalue which cannot be assigned, in VS C++ 2010 this code will produce following error “error C2106: ‘=’ : left operand must be l-value”.
3* – example of expression that yields lvalue
4* – here we have a function returning lvalue which can be assigned
5* – here function returns rvalue which cannot be assigned and results in the same error like in 2*

From those examples it is easy to see that lvalue can be used where rvalue is required, but not the other way around, rvalue cannot be used in place of lvalue with the exception when c++11 new construct – rvalue reference is used.

Fundamental property of lvalue is that it has a location. This location can be used in assignment expressions, you can use lvalue to take its address.

lvalues are also said to be persistent, rvalues on the other hand are temporary, or ephemeral and are available only until end of full expression, but if rvalue it assigned to const reference or pointer then its lifetime will be prolonged.

lvalues not always can be modified, for example when they are const variables or array types.

Now lets introduce xvalue, glvalue and prvalue. With C++11 comes new language feature called rvalue references. It allows to steal internals of rvalues and use them in constructing new objects that can be later on used as lvalues. This was actually possible in C++03 but was largely complicated. Those three new expression categories were introduced to properly describe this new language feature.

Xvalue (eXpiring value) is the most important new concept, it is tightly connected to new move semantics. To shortly describe move semantics, lets see an example:

In C++03 you would initialize class object with string as follows:

class MyData {
std::string s;
public:
  MyData(std::string ss)
   :  s(ss) // (2*) here creates another allocation and copy
  {}
};

MyData md("mydata"); // (1*) creates one temporary which requires at least one allocation

The problem is that it introduces temporary rvalues that allocate memory and also do some other operations. With C++11 comes possibility to actually “steal” inner contents from such temporary in (2*) and eliminate not needed allocations/operations:

class MyData {
std::string s;
public:
  MyData(std::string ss) : s(std::move(ss)) {} // here no allocations are created, inner guts of temporary ss are used to initialize s variable
};

MyData md("mydata"); // here creates temporary string object

Xvalue in the second example is the std::move(ss) experssion, it returns rvalue reference which is an indication to std::string class that we want to steal inner contents from this temporary and use it in s variable. What std::move really do is to turn named variable into unnamed reference, this is an indication to compiler that we really know what we are doing, and we want s variable to be constructed from ss using move semantics. So the new name for such expression was needed, since move semantics in this example is explicit – without std::move we are back in C++03 world.

After that ss in MyData() constructor is actually empty, standard guarantees that such objects are in a “valid but unspecified state”, that means you can assign new values to it, but current value is not defined.

So – since C++11 rvalue got new semantic which is being an rvalue reference – this means that besides its old semantic it can also behave as xvalue described above.

Now to those two last names:
– when you see prvalue (“pure” rvalues) – this indicated plain old rvalues from C++03. Those are not xvalues.
– glvalue (“generalized” lvalue) is just a grouping for lvalues and xvalues.

[refs]
http://www.cprogramming.com/c++11/rvalue-references-and-move-semantics-in-c++11.html
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3055.pdf

RandomAccessFile does not lock your file by default

If you use this class for writing then you probably want to ensure that you are the only one who is allowed to modify given file. I recently was fixing code that used RandomAccessFile for writing huge files downloaded from internet, but it didnt lock those file during writing. Whats wrong with that? Actually from the user perspective nothing, those file wont be accessed any way by any other software, but what we are missing here is the fact that only one thread should be allowed to read/write those files is an invariant which should be checked in code to trap any errors that could show up. And if this invariant were checked in the code I was fixing, developer would find his bug much earlier.

In todays heavily multithreaded environments it is easy to introduce errors, thats why invariants should be checked whether possible and exceptions should be thrown always when they are not correctly met.

Below is a small method for checking if file is locked or not.

public static boolean isFileLocked(File f) {
	RandomAccessFile file = null;
	FileChannel fileChannel = null;
	FileLock fileLock = null;

	try {
		file = new RandomAccessFile(f, "r");
		fileChannel = file.getChannel();
		if (fileChannel == null)
			return false;
		fileLock = fileChannel.lock();
	} catch (FileNotFoundException fnfe) {
		return false;
	} catch (Exception e) {
		// ClosedChannelException
		// NonWritableChannelException
		// OverlappingFileLockException
		// FileLockInterruptionException
		// AsynchronousCloseException
		// IOException
		return true;
	} finally {
		try {
			if (fileLock != null)
				fileLock.release();
			if (fileChannel != null)
				fileChannel.close();
			if (file != null)
				file.close();
		} catch (IOException ioe) {
			return false;
		}
	}
	return false;
}

Finding buffer overruns and other heap corruptions under Windows

Windows SDK comes with great tool called debugging tools – it includes small application called gflags that allows to quickly find all buffer overruns and reads of unallocated memory. Actually I would really like to see it integrated in Visual Studio IDE. So how it works. First you register your application with gflags.exe, and then run it. During program testing, system will throw exceptions each time you do something bad with memory (like read or write unallocated regions). Gflags during each allocation will allocate slightly more memory and mark it as non-commited, so any access to it will result in exception which can be easily cought by Visual Studio – yes, after registration with gflags, you can run your program under VS debugger and catch all exceptions, read callstack and all variable contents.

Here is how you register your application with gflags:

gflags /p /enable pheap-buggy.exe /full
(http://msdn.microsoft.com/en-us/library/windows/hardware/ff543097%28v=vs.85%29.aspx)

so even if your program runs flawlessly, use gflags to find bugs in regions which you would never suspect

Background Task Sample in C++ Metro Style App

In this post I will show step by step how to write metro style app that will register for system notifications. In samples section for Metro style apps in MSDN dev center, you can find “Background Task Sample” sample code for C# and JavaScript. There is also a white paper for Background tasks at this link:

http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=27411

One thing I was not able to reproduce from this document is putting Run() method inside main app.

Sample app described below registers background task to Internet availability event. You can easily change it to TimerTrigger, but testing it is very difficult since you must wait 15 minutes before it will be notified to you. I have also found that TimeZoneChange event is easy to trigger manually, you just have to change your time zone.

I assume you have installed Windows Developer Preview

1. Open Visual Studio Express 11  and create new project : “Visual C++” -> “Windows Metro Style” -> Application. Name it BackgroundSample.

2. Now add WinRT Component DLL. Choose File -> New Project -> “Visual C++” -> “Windows Metro Style” -> “WinRT Component DLL”. Name it BackgroundTask, and choose “Add to solution” in Solution combobox.

3. In solution explorer right click on BackgroundTaskSample project and choose Project Dependencies. Then check “BackgroundTask” from project list.

4. Now you have to reference your WinRT component in main app. Once again right click on BackgroundSample project and choose References… Click on Add New Reference, and once new window shows up, double click on BackgroundTask item. There should appear a green tick on the left of it.

5. Now we need to setup Background Task declaration in manifest file. Double click on Package.appxmanifest file in BackgroundSample project file list. Then:

– from Application UI tab change “Toast Capable:” from No to Yes. We will use toast to inform of background task completion.

– make sure “Internet (client)” capability is checked on Capabilities tab

– On Declarations tab choose “Background Tasks” from available declarations combobox, and click Add. Then check SystemEvent and Timer. Change EntryPoint to: BackgroundTask.WinRTComponent.

6. Ok, now lets start coding. Open MainPage.xaml.cpp file in you main app, and add in MainPage::MainPage following code (below using namespace code block):

using namespace Windows::UI::Xaml;
using namespace Windows::UI::Xaml::Controls;
using namespace Windows::UI::Xaml::Data;
using namespace Windows::ApplicationModel::Background;
using namespace Windows::Data::Xml::Dom;
using namespace Windows::UI::Notifications;

ToastNotifier^ toastNotifier = nullptr; // Ugly global variable
void DisplayToastWithImage(String^ str) {
  XmlDocument^ toastXml =
    ToastNotificationManager::GetTemplateContent(ToastTemplateType::ToastText01);
  XmlNodeList^ textElements = toastXml->GetElementsByTagName("text");
  for (int i = 0; i Length; i++) {
    XmlText^ newNode = toastXml->CreateTextNode(str);
    textElements->Item(i)->AppendChild((IXmlNode^)newNode);
  }
  ToastNotification^ toast = ref new ToastNotification(toastXml);
  if ( toastNotifier == nullptr )
    toastNotifier = ToastNotificationManager::CreateToastNotifier();
    toastNotifier->Show(toast);
}

in MainPage::MainPage() add following code below InitializeComponent():

/// If task was already registered then only add Completed event handler
/// System remembers registrations even if app was closed.
bool registrationIsRequired = true;
auto iter = BackgroundTaskRegistration::AllTasks->First();
if ( iter->HasCurrent ) {
  do {
    auto guid = iter->Current->Key;
    IBackgroundTaskRegistration^ task = iter->Current->Value;
    registrationIsRequired = false;
    DisplayToastWithImage("Already registered");
    task->Completed += ref new BackgroundTaskCompletedEventHandler(this, &MainPage::OnCompleted);
    //task->Unregister(true);
    } while ( iter->MoveNext() );
}

if ( registrationIsRequired ) {
  BackgroundTaskBuilder^ myTaskBuilder = ref new BackgroundTaskBuilder();
  ISystemTrigger^ trigger = ref new SystemTrigger(SystemTriggerType::InternetAvailable, false);
  myTaskBuilder->SetTrigger(trigger);
  myTaskBuilder->TaskEntryPoint = "BackgroundTask.WinRTComponent";
  myTaskBuilder->Name = "InternetAvailableChange";
  IBackgroundTaskRegistration^ taskReg = myTaskBuilder->Register();
  taskReg->Completed += ref new BackgroundTaskCompletedEventHandler(this, &MainPage::OnCompleted);
  DisplayToastWithImage("Registered");
}

Now add OnCompleted method (and appropriate declaration in MainPage header file):

void MainPage::OnCompleted(Windows::ApplicationModel::Background::IBackgroundTaskRegistration^ sender, Windows::ApplicationModel::Background::BackgroundTaskCompletedEventArgs^ args) {
  /// This code will be called even if App is closed.
  DisplayToastWithImage("MainPage::OnCompleted");
}

Whats left is to modify WinRT Component project:

add

using namespace Windows::ApplicationModel::Background;

in WinRTComponent.h file, then derive WinRTComponent class from IBackgroundTask and implement Run method:

public: virtual void Run(IBackgroundTaskInstance^ taskInstance) override {
}

it is empty, because we show Toast in OnCompleted method in main app.

…thats all, after running this app, you can turn off your network card and again on, and a “MainPage::OnCompleted” toast will show up.

Below you will find attached sample project:

BackgroundTaskSample.zip

Moving to PreferenceActivity

PreferenceActivity seems to be the best choice when it comes to allowing user edit application settings. If you started with custom dialog with lots of widgets for your preferences and now you want to move to PreferenceActivity then be prepared for some difficulties:

– your Shared Preferences keys must be of String type and values must be of String or Boolean types.
– if you have any int values in your preferences, then remember to convert it to string before PreferenceActivity will touch it, otherwise be prepared for exceptions
– you should not start PreferenceActivity derived Activity class using sub-activities, but rather use SharedPreferences.OnSharedPreferenceChangeListener and apply changes once preference changes, this might cause troubles if processing is time consuming
– if you used arrays.xml for your widget allowed values (ie. spinner), then make sure that allowed values are strings, ie.if you had 3 then change it to “3”. PreferenceActivity works with Strings – remember.
– and remember to read docs on PreferenceActivity, since its methods are heavily deprecated since HONEYCOMB release

Using private/public key for authentication in Putty

If you want to make your life easier and no longer need to enter password, follow these steps:

1. Use PuttyGen to generate public/private keys and save them to local folder as my_key.ppk and my_key.pub.
2. Store my_key.ppk to ~\.ssh\authorized_keys2 file, remember to convert it to openssh format:
– remove two first lines and one last line
– add ssh-rsa followed with one space
3. In putty specify my_key.ppk in Connection->SSH->Auth setting
4. You might also need to:
$ chmod 600 ~/.ssh/authorized_keys2
$ chmod 700 ~/.ssh/
on server

You might also add my_key.ppk to Putty Pageant which will provide your key automatically to Putty.

Now if all is working ok, you might want to add your login username to Conenction->Data->Auto-login username editbox. No more username/login prompts.

This procedure is required by some other software like TortoiseGIT.

Advanced serialization tips

Seralization knowledge not fully covered by official training kit.  These are actually notes from CLR via C# chapter 24.

1. SoapFormatter as of .NET 3.5 is obsolete, for production code use XmlSerializer or DataContractSerializer.
2. Instances can be deep cloned using binary serialization, remember to set:
bin_formatter.Context = new StreamingContext(StreamingContextStates.Clone);
3. You can serialize multiple objects to single stream
4. If You serialize instances of objects which are located in custom loaded assemblies (using LoadFrom), then SerializationExceptin might be thrown. To load assembly when requested, implement delegate that matches System.ResolveEventHandler and register this method with System.AppDomain’s AssemblyResolve event just before deserialization and remove it after it.
5. If some of the objects in the graph are not serializable, then exception will be thrown. It is thrown not before serialization, it can happen any time during it. This can cause resulting stream to contain corrupt data. In order to fail gracefully, serialize to memory stream, and if all is ok, send it to network stream etc.
6. The SerializableAttribute custom attribute may be applied to reference types (class), value types (struct), enumerated types (enum), and delegate types (delegate) only. But since enumerated types and delegates are serializable by default, it makes no sense to add this attribute to them.
7. Serializable attribute is not being inherited, if super class is non serializable then subclasses are not either.
8. Do not set Serializable to automatically generated properties, their names can change during compilation, making it not possible to deserialize.
9. Formatters makes use of FormatterServices class for serialization and deserialization, steps are as follows:

Serialization:
– Call FormatterServices.GetSerializableMembers to get MemberInfo[].
– Then call FormatterServices.GetObjectData to get array of values for each MemberInfo instance in the above array.
– Store assembly identity and type’s full name.
– Store each member, their type, name and value.

Deserialization:
– Load assembly identity information and if required load it. Call FormatterServices.GetTypeFromAssembly. This call returns System.Type that represents object to be deserialized.
– Call FormatterServices.GetUninitializedObject with previously returned System.Type, this returns an non-constructed object, with fields set to zero values.
– Now call FormatterServices.GetSerializableMembers to get MemberInfo[]. Those members will need to be deserialized.
– Formatter creates object data from the stream, those will be values of our members.
– With all prepared data call FormatterServices.PopulateObjectMembers, it requires reference to object, MemberInfo[] and Object[] – with values (deserialized in step above).

10. Implementation of ISerializable interface (to get freedom of how serialization happens and get rid of slow reflection), requires implementing its methods in each subclass, and also calling in them, super class versions.
11. Always add followng attribute to your ISerializable.GetObjectdata: [SecurityPermissionAttribute(SecurityAction.Demand, SerializationFormatter = true)]. This will prevent outside code from using this method in some malicious ways.
12. When using SerializationInfo class in c-tor, you should call the same Get* methods types as in ISerializable.GetObjectData. If there is a mismatch when deserializing given data, framework will try to convert it with IFormatterConverter, which internally calls System.Convert for core types. For other types it uses type’s IConvertible implementation.
13. SerializationInfo also provides enumerator, which returns SerializationEntry instances.
14. To custom serialize class which base type does not implement ISerializable, use this.GetType().BaseType, and serialize/deserialize it manually. This might be not possible with private fields.
15. Serializing a Type as a different Type, and Deserializing as a different object. This is of use when using singletons, DBNull object. For serialization:
– Add to ISerialization.GetObjectData, SerializationInfo.SetType() method call. As a parameter specify type of custom class that derives from IObjectReference.
16. Serialization surrogates: this is a technique that allows to serialize types not designed for serialization, to deserialize type to a new version, etc.. It is realized by createing class that implements ISerializationSurrogate interface. Its methods GetObjectData and SetObjectData are used for serialization and deserialization. In orderd to let Formatter know about our serializing surrogate implementation, create and initilize SurrogateSelector class and set it to Formatters SurrogateSelector property.
17. The BinaryFormatter class has a bug that prevents a surrogate from serializing objects with references to each other. For fix look into CLR Via C#.
18. Finally there is a technique to deserialize data to a different object than the one used during serialization. This is of use when new version of type is created, or type is moved to other assembly, client want to deserialize data from server to different local type. To use it implement SerializationBinder abstract class, and set it to Formatter’s Binder property.

Below are notes from other sources:

19. ObjectManager tracks deserialized objects. Formatter uses it to check if given reference is backward reference – its object was already deserialized, or it is forward reference – instance was not yet deserialized – this requires recording fixup with ObjectManager. Fixup means that reference will be assigned correct object in the future when actual object will get deserialized.
20. Marshaling is the process of packaging and sending remote procedure calls across process and application boundaries using serialization and deserialization.

Coloring C++ code for bloging

Finally got it working  (as seen in previous post). You need Visual Studio (2005 – 2008), and CopySourceAsHtml addin. Actually it might show exceptions and you might need to copy as HTML 2-3 times before it will work. Also after pasting it to wordpress in new post, do not change to Visual, stay with HTML, otherwise your code will get messed up.

not perfect but still it gives best results

[edit] forget about using external tools and use hints from: http://en.support.wordpress.com/code/