The origin of the name Sufi probably goes back to the term suf, which refers to imple woolen cloaks. The original Sufi were basically mystics - people who followed a pious form of Islam and who believed that a direct, personal experience of God could be achieved through meditation and self-discipline.
Sufism is important to the development of Islam because it is in this tradition that the more spiritual and mystical aspects were preserved. This stands in contrast to the mainstream of Islam which, through its first centuries, was more concerned with the expansion and organization of the general community.
Over time, of course, Sufism became institutionalized, and individual mystics attracted groups of students and followers who learned to follow the mystic's path, or tariqa. The tariqa is a set of mental and even physical exercises designed to aid in the development of communion with God.
Eventually, the term tariqa came to refer to any group of people following a Sufi mystic, or shaikh (which means "old man" or "teacher"). Very often such groups took on something of a monastic organization, not unlike Buddhist groups following a teacher.
Sufi orders or brotherhoods came to be called turuq, although it was not until the 12th and 13th centuries that they became stable enough to survive the death of their founder. This was achieved by the founder personally nominating a successor, and thus there are some today which can trace their lineage back to that time.
Some Sufi mystics did not even retain theism in their conception of Islam: they substituted the Realm of Truth for Allah. When Buddhist influences reached Iraq, some Sufis there moved close to outright atheism (as did some zindiq or free-thinking sufis), and emphasized self-annhilation as the entire goal.
This, however, was uncommon. Whereas mainstream Islam tended to be rather scholastic in nature, Sufism focused on personal peity, which was easier for many Muslims to accept and live by. Although the Sufis tended to integrate beliefs and practices which came from outside Islam, those beliefs and practices were integral in the cultures to which Islam had been transplanted.
This made Sufism more appealing to indigenous people converting to Islam. Thus, Sufis were effectively Muslim missionaries, presenting to new peoples a more interesting face and belief system. They also tended to act as defenders of the lower classes against corrupt rulers.
Shi'ite beliefs had a lot of influence on Sufism, but mostly in the early days when Shi'a was more a school of thought than a distinct sect with its own ideology. One important idea which was transmitted from from the Shi'a to Sufism is that of the Mahdi, a messianic figure which will come to save Muslims from corrupt worldly rulers.
For the Shi'a, the Mahdi is the Twelfth Imam, who is currently hidden. For the Sufi, however, the Mahdi won't necessarily be a descendant of Ali, but will instead be a divinely guided leader who will appear at the End of Time to restore justice and truth over the forces of the indfidels. Although this has never been a formal doctrine in Sunni Islam, it has gained a foothold in the imaginations of most Muslims thanks to the Sufis.
Mainstream Islam has regarded Sufism with some suspicion because of the extremism in beliefs which mysticism is prone to. For example, the mystic al-Hallaj was executed in the ninth century because his statement "I am the Truth" was seen as a self-identification with God.
Nevertheless, such force was not the norm, and most Muslim rulers tried to control rather than eliminate Sufism - a good idea, considering how strong its following was among the common people. Thus, many religious leaders attempted to create some sort of synthesis between Sufism and orthodox Sunni Islam. A common method was to assert that Sufi mysticism is not a way to learn any new facts about reality which were not in the Qur'an, but is a valid way of understanding reality.-->