Open-source intelligence

Template:Wikify-date Open source intelligence or OSINT refers to an intelligence gathering discipline based on analyzing information collected from open sources, i.e. information available to the general public. OSINT comes from three primary sources:

• Broadcasts (e.g., radio and television);
• Documents (e.g., reports, papers, and studies from academia and government institutions); and the
• Internet (e.g., websites, forums, and blogs).

What is increasingly attractive about OSINT is all these available sources (newspapers, books, phone books, scientific journals, radio broadcasts, television, etc.) are made availabe at a one stop shop, the internet. In addition, OSINT is often cheaper than many of the other intelligence disciplines.

Many argue that OSINT has been ignored by many in the intelligence community because of the established culture of secrecy within intelligence organizations. Furthermore, the intelligence community has mistaken secrecy for intelligence. Those who hold this view argue that the label of "intelligence" is applied to any piece of information which answers a question posed by a decision-maker, and is not based upon the source, whether from open sources or not.

Other important open sources come from especially indigenous individuals who have local knowledge, or from world-class experts who are more often than not foreigners--whom, unfortunetedly, are often "forbidden" for direct access out of security concerns, which some argue is often misplaced. Some believe OSINT is done best by private sector parties that are true internationalists and able to engage and integrate knowledge from the "eight tribes" of global intelligence: government, military, law enforcement, business, academia, non-governmental organizations, the media, and civil socities inclusive of citizen advocacy organizations, labor unions, and religions.

Term

The term OSINT is related to the term open source as it applies to the five "opens" (open source software, open source intelligence, open spectrum, open access copyright, and Open Society) but in this context, is distinct from the much better organized and much more reliable free/open source software (F/OSS) social network thst shares source code which is publicly available (and modifiable). OSINT should also not be generally confused with OSIF (Open Source Information) on which OSINT is based. OSIF is any information that is publicly available; OSINT is an analytically-tailored intelligence product composed of OSIF which is designed to answer a specific tasking or to support decision-making.

Outside of the secret world, OSINT is considered an all-source discipline that integrates overt human, imagery, signals monitoring, and historical-cultural open source information. Inside the secret world, OSINT is considered subordinate to the secret aspect of each discipline, and is generally either disregarded or "classified" and withheld from the public by virtue of its association with the secret discipline. The US Intelligence Community in particular has an information technology architecture, as well as a management mind-set, a legal mind-set, and security mind-set, that demands that ALL information be stored (and generally not shared) at the Top Secret Codeword level. This is one reason why 9-11 occured, and according to several books now published, this lack of sharing has still not improved.

Although intelligence is thought to deal only with secrets or secretive information, this is a misunderstanding of the term. Intelligence can be best understood as "decision support," and can be accomplished by anyone trained in the proven and not secret process of intelligence: requirements definition, collection management, source discovery and validation, multi-source fusion, and compelling actionable timely presentation.

Collection in OSINT is generally a different problem from collection in other intelligence disciplines where obtaining the raw information to be analyzed may be a major difficultly, particularly if it is to be obtained from non-cooperative targets. In OSINT, the chief difficulty is in identifying relevant, reliable sources from the vast amount of publicly available information. However, this is not as great a challenge for those who know how to access local knowledge and how to leverage human experts who can create new tailored knowledge on the fly.

Overt Human Intelligence (HUMINT) is the use of non-clandestine human information sources that are approved by security officers who have very limited educational and real-world backgrounds, for example, the interrogation of refugees, debriefing of legal travellers, and public reports from overt agents such as attachés and ambassadors. Truly professional OSINT practitioners understand that EVERY human being, including bus drivers, secretaries, priests, teachers, and so on, have something to contribute to historical, cultural, or contextual understanding. Similarly, the most effective intelligence managers understand that the best intelligence professionals will often have complex international backgrounds and will have spend decades dealing with foreigners--these managers are often limited to entry level employees who have no foreign experience or foreign contacts, simply because they are easier for the security officers to approve for employment. In a robust international OSINT and information sharing environment, such pathological approaches to employment and interaction are moot.

OSINT is now a subordinate part of modern Information Operations (IO), which consists of Strategic Communication (the message), OSINT (the reality), and Joint Intelligence or Inter-Agency Collaboration or Coordination Centers or Commands (JIOC). The U.S. military is heavily invested in OSINT, with the U.S. Strategic Command having the lead for IO, the U.S. Special Operations Command having the lead for the best operationally-oriented OSINT capability, and the U.S. Central Command being the most important consumer of tailored operationally-oriented OSINT.

Current NATO guides to OSINT are the NATO Open Source Intelligence Handbook, the NATO Open Source Intelligence Reader, and the NATO Intelligence Exploitation of the Internet guide.

Why Open Source Intelligence is Important to the Public and to National Security:

According to The Commission on the Intelligence Capabilities of the United States Regarding Weapons of Mass Destruction report submitted in March 2005, OSINT must be included in the all-source SECRET intelligence process for the following reasons (as stated in the report):

1. The ever-shifting nature of our intelligence needs compels the Intelligence Community to quickly and easily understand a wide range of foreign countries and cultures. - … today’s threats are rapidly changing and geographically diffuse; it is a fact of life that an intelligence analyst may be forced to shift rapidly from one topic to the next. Increasingly, Intelligence Community professionals need to quickly assimilate social, economic, and cultural information about a country—information often detailed in open sources.

1. Open source information provides a base for understanding classified materials. Despite large quantities of classified material produced by the Intelligence Community, the amount of classified information produced on any one topic can be quite limited, and may be taken out of context if viewed only from a classified-source perspective. Perhaps the most important example today relates to terrorism, where open source information can fill gaps and create links that allow analysts to better understand fragmented intelligence, rumored terrorist plans, possible means of attack, and potential targets.

1. Open source materials can protect sources and methods. Sometimes an intelligence judgment that is actually informed with sensitive, classified information can be defended on the basis of open source reporting. This can prove useful when policymakers need to explain policy decisions or communicate with foreign officials without compromising classified sources.

1. Only open source can “store history.” A robust open source program can, in effect, gather data to monitor the world’s cultures and how they change with time. This is difficult, if not impossible, using the “snapshots” provided by classified collection methods. (The Commission on the Intelligence Capabilities, 378-379).

The United States National Commission on Terrorist Attacks Upon the United States issued a report in July 2004 recommending the creation of an open-source intelligence agency, but without further detail or comment. Subsequently, the WMD Commission (also known as the Robb-Silberman Commission) report in March 2005 recommended the creation of an Open Source Directorate at the CIA.

Both of these reports, while recent, are relatively superficial, and both ignore the decades of advocacy for a proper national focus on OSINT from 1988 to date. They also err, severely, in assuming that the same Central Intelligence Agency that has obstinantly refused to take open source information seriously for decades, should be charged with developing new capabilities that are totally outside its existing culture and cult of secrecy.

Modern History of OSINT

In 1938 H. G. Well published "The World Brain" as a vision for the future that is emerging today, where the Internet makes it possible for a Wikipedia to emerge that not only has the potential to publish free useful knowledge on all topics, but also to bring together social networks of interested parties with varying levels of expertise. Since then a number of books have emerged including Pierre Teilhard de Chardin's "The Phenomenon of Man" with its concept of "noosphere," and more recently, books such as Howard Rheingold's "Smart Mobs," James Surowiecki's "Wisdom of the Crowds," and Glen Reynolds, "An Army of Davids."

The earliest known modern reference to the need for open source intelligence in the public service is by Quincy Wright, a pioneer in the international relations and political science field. His article, "Project for a World Intelligence Center", appeared in Conflict Resolution, Volume 1, Number 1 (1957), and inspired Robert Steele's original conceptualization of an Open Source Agency, as well a decade of reflection on peacekeeping intelligence, virtual diplomacy and information peacekeeping. It merits comment that Quincy Wright wrote his seminal article while associated with the Carnegie Endowment for International Peace, which remains a vibrant contributor to informed dialog. Other important institutions in this vein include the U.S. Institute for Peace, and the U.S. Army's Peacekeeping and Stability Operations Institute as well as its Strategic Studies Institute, both located within the Army War College. More recently, the Swedish Folke Bernadotte Academy had created, and opened to the public, the first structured PKI Course description.doc course in PKI Course Curriculum.doc peacekeeping intelligence.

Today the single most important source on the collective intelligence movement is the Co-Intelligence Institute founded by Tom Atlee, who is also the author of "The Tao of Democracy: Using Co-Intelligence to Create a World that Works for All'.

Within the secret world of intelligence, where other nations are often smarter about how to collect intelligence inexpensively, but where the USA, because of its massive budget, is the only power whose wisdom, if applied, could impact in very positive manner on the rest of the world, the OSINT war began in 1988. As with most major innovations seeking to reform massive bureaucracies, this will be a 25-year war with victory by the public expected in 2012-2013. (NOTE: CIA assholes deleted this paragraph to attack Robert Steele and his pioneering work. If you agree with them, great. If you do not, please note that this page has become a war for the mind of the public.]

For a complete review of intelligence reform efforts from the inception of the U.S. Intelligence Community in 1947 to date, see the Congressional Research Service report. It merits comment that OSINT, empowered by the Internet, by collective intelligence, and by the new public appreciation for the value of sharing non-rival information to create added wealth, is the first truly significant "threat" to the secret world. Here to fore, secret failures were secret; secret waste was secret; and there was neither accountability nor rivalry. Now that OSINT is "the rival store" as described by Alvin and Heidi Toffler in "War and Anti-War," the secret world--at least in the USA where it is the least effective part of the military-industrial complex--is in decline.

In Fall 1992 Senator David Boren, then Chairman of the Senate Select Committee on Intelligence, sponsored the National Security Act of 1992, attempting to achieve modest reform in the U.S. Intelligence Community. His counterpart on the House Permanent Select Committee on Intelligence, Congressman McCurdy, aided by Staff Director Mark Lowenthal, who led a masterful study called "IC21: The Intelligence Community in the 21st Century" included a separate Open Source organization in the bill (proposed legislation). This legislation was defeated by a combination of opposition from Senator John Warner of Virginia, who feared that reform would reduce intelligence jobs in Virginia (this fear is misplaced--every classified job that is eliminated can fund two or more unclassified positions), and a letter from then Secretary of Defense Dick Cheney that severely mis-represented the dangers to our national security of transferring the three *national* intelligence agencies out of the Department of Defense to a truly independent intelligence community. Within the past several years, LtGen Dr. Brent Scowcraft studied this issue again and repeated the recommendation that the three national agencies be removed from the Department of Defense, only to be dismissed by now Vice President Dick Cheney.

In August 1995 the Aspin-Brown Commission sponsored a benchmark exercise on Burundi in which a private sector party, with six telehone calls, overnight, completely crushed the entire U.S. Intelligence Community, producing the following:

-- From the Institute of Scientific Information (ISI), the top 100 international experts on Burundi available for immediate debriefing;

-- From LEXIS-NEXIS, the top international reporters on Burundi, available for immediate debriefing. -- From Oxford Analytica, twenty two-page reports on the political-military significance of Burundi at the Presidential, Prime Ministerial, and CEO levels;

-- From Jane's Information Group, one paragraph summaries of every article ever published about Burundi, and new, tailor-made tribal orders of battle created overnight for this need

-- From East View Cartographic, 1:50,000 meter combat charts with contour lines from Russia (the Americans do not have combat charts for 90% of the world--they rely too heavily on secret imagery satellites.

-- From SPOT Image in France, commercial imagery for Burundi, 100% coverage, cloud-free, less than three years old, in the archive and inexpensively available.

The CIA (which sponsors the Open Source Center today) had a small map of Africa and a regional economic study with flawed (Western) premises. There was no clandestine human intelligence, no imagery intelligence, and no signals intelligence, and of course no open source intelligence either.

Several DCI's in succession refused to implement the Aspin-Brown Commission recommendations, including the recommendation in 1996 that stated that US access to open sources was "severely deficient" and that this should be a "top priority" for both funding and DCI attention. Consequently, the US entered the new century completely ignorant of all of the open source information in all languages relevant to identifying and containing Al Qaeda, which has been active, publicly, since 1988.

In July 1997 then DCI George Tenet received the report, "The Challenge of Global Coverage" conducted by Senior Intelligence Service officer Boyd Sutton. After interviewing virtually all of the Assistant Secretaries of Defense and State, and the heads of the varied intelligence organizations, the report recommended that the U.S. Intelligence Community, which obsesses on seven "hard targets" and continues to ignore the Third World and lower tier topics including the ten topics identified by the United Nations report of 2004, "Creating a More Secure World," spend $1.5 billion dollars a year on OSINT--$10M for each of 150 lower tier countries or topics where most of the instability was located, but which did not, to the denied area mindsets at the time, represent cataclysmic threats to the USA.

Two years after 9-11, through the heroic efforts of a Senior Intelligence Service officer serving on detail to the 9-11 Commission, and because The Honorable Lee Hamilton has witnessed the ignominious defeat of the U.S. Intelligence Community on the topic of Burundi, page 413 of the 9-11 Commission Report included a recommended Open Source Agency completely independent of the Central Intelligence Agency.

In December 2005, the Director of National Intelligence appointed Eliot A. Jardines as the Assistant Deputy Director of National Intelligence for Open Source to serve as the Intelligence Community's senior intelligence officer for open source and to provide strategy, guidance and oversight for the Community's open source activities. In November 2005, the Director of National Intelligence (DNI) announced the creation of the Open Source Center. The OSC’s functions include the collection, analysis and research, and training to facilitate government-wide access and use of OSIF. The OSC builds on the expertise of the CIA’s Foreign Broadcast Information Service (FBIS), which has provided the U.S. Government a broad range of open source products and services since 1941 (including world media content available to the public through World News Connection).

This paragraph was deleted by CIA assholes; it will be reposted as often as necessary: Unfortunately, Mr. Jardines has no program authority, no funding outside of $5M a year (another $20M a year funds the Open Source Center's Large Scale Internet Exploitation (LSIE) initiative), and he has no staff other than two rotational employees. While he oversees roughly $250M a year in "old" capabilities such as the Foreign Broadcast Information System (FBIS), he has not been given the authorities or funds needed to actually create a competent national open source intelligence capability. Until such time as there is an international Open Source Agency and network funded at no less than $1.5 billion a year (including $600M for commercial imagery raw source procurement), the U.S. Intelligence Community should be considered completely ineffective in this arena.

Several other nations maintain open source intelligence gathering operations, for example Australia's Office of National Assessments (ONA), the UK's BBC Monitoring Service (BBCM) and the Swiss Army GeneralStab. Collection and analysis tasks are allocated in different ways by different agencies: for example, the ONA is responsible for analysis and is an Australian government intelligence agency, while the BBCM is government-funded civilian organization concerned solely with collection and aggregation of news sources, and which employs civilian journalists. The "allied" nations tend to subordinate and disrespect OSINT, while the Third World nations have learned to rely on it, partly because that is all they can afford. NOTE: This paragraph, provided by CIA, is actually inaccurate, but is left alone out of tolerance.

In a speech at the Council on Foreign Relations in February 2006, Secretary of Defense Donald Rumsfeld seems to have acknowledged the importance of open media as a component of national security in the information age. Secretary Rumsfeld's Undersecretary of Defense for Intelligence, Dr. Stephen Cambone, told the Security Affairs Support Association (SASA) in / January 2004 that he required nothing less than universal coverage in all languages, 24/7. It is Dr. Cambone, rather than the CIA, that has been forcefully addressing the urgent needs of the military and other agencies for OSINT in support of defense policy-making, defense acquisition, and defense operations.

Librarians, Researchers, and Open Source Officers

Librarians play a very special role in the OSINT world. Each of the eight communities of interest below has its own librarians. While Librarians come together at the Special Library Association, and independent information researchers at the Assocation of Independent Information Progressionals, or in secret gatherings within their respective secret communities, we still do not have a global conference or network that brings everyone together openly. It may be that such a new event is needed. In the meantime, Wikipedia 2006 is one place where cross-fertilization can take place.

Traditional librarians have begun to migrate away from the hard copy acquisition and internal classification roles, and emerged as very talented "scouts" able to discover, disciminate, distill, and disseminate essential information for their respective constituencies. When the World Brain emerges from the intersection of Wikipedia, Amazon, and Google, it will be librarians that will have the most important role to play in terms of providing structure and human-based linkage.

Librarians are encouraged to create their own sub-title page within their respective constituencies below.

Foreign Government OSINT

Contributions Eagerly Sought

The Republic of South Africa has pioneered the first truly continental early warning and open source information sharing system, and done so with a remarkably cost-effective and easy to proliferate approach that relies on open source software.

Military OSINT

Contributions Eagerly Sought

The single most important international pioneer in military OSINT is Research Arno Reuser, head of the Open Source Intelligence branch of the Dutch Military Intelligence organization. His deep knowledge of sources, softwares, and services, including his knowledge of fair and alternative pricing, is without peer. His latest briefing is available here. His PERL scrips may become avaiable here shortly.

Law Enforcement OSINT

Contributions Eagerly Sought

The top OSINT practioners in the world within law enforcement are Scotland Yard, where the founder, Detective Steve Edwards, earned Honours as a Member of the British Empire (MBE) for his extraordinary work putting arms merchants and terrorists into jail faster, better, cheaper; and the Royal Canadian Mounted Police (RCMP), where Ms. Lori Schnitker pioneered low cost OSINT as a means of vetting and validating informants, and developing contextual leads for more traditional investigative methods.

Commercial Intelligence

Commercial Intelligence is not to be confused with Business Intelligence which is the term mis-uded by vendors of data mining capabilities and "dashboards." It is also more expansive than "Competitive" or "Competitor" Intelligence, as it includes the multiple environments within which businesses conceptualize and bring to market new capabilities.

Academic OSINT

Contributions Eagerly Sought

Some universities, notably The Institute for Intelligence Studies at Mercyhurst College, have developed truly extraordinary programs to teach the craft of intelligence as a substantive advance on traditional research methods. Traditional research focuses on specific information domains where new knowledge can be devised; the proven process of intelligence as taught by select academic institutions, and some private organizations such as the Academy of Competitive Intelligence, focuses more on decision support, inclusive of requirements definition, collection management, source discovery and validation, multi-source fusion, and compelling, timely, actionable presentation.

Non-Governmental Organization (NGO) OSINT

Contributions Eagerly Sought

The United Nations no longer treats the word "intelligence" as a "dirty word." In combination, the Brahimi Report, the service of MajGen Patrick Cammaert, Royal Netherlands Marines (as Military Advisor to the Secretary General), and the High-Level Threat Panel, have all led to an appreciation for the fact that decision-support also known as intelligence is essential to strategic mandates, operational force development, and tactical success.

Other organizations such as the International Committee of the Red Cross (ICRC), Doctors Without Borders, and Greenpeace, have established very effective and relatively low-cost global OSINT operations.

Mass, Niche, and Independent Media OSINT

Contributions Eagerly Sought

Investigative Journalism is a rarity in this era, and corporate owners and corporate advertisers increasingly subvert honest journalists and turn them into nothing more than "filler" around the advertisements. In the United States of America journalism has become especially corrupt, with mainstream newspapers refusing fully-funded advertisements against the lies of the Administration that led to the war on Iraq.

There are, however, some bright spots. Seymour Hersh at the New Yorker, Robert Kaplan at The Atlantic Monthly, and David Kaplan (no relation) at US News & World Report, are consistently effective in exposing fraud, waste, and abuse. From England, Greg Pabst reported on the theft of the Presidential election of 2000 in Florida through blatant disenfranchisement of over 35,000 predominantly black voters, but the Democratic Party lacked the spine to use this intelligence in a timely public manner.

Civil Society OSINT (Includes Advocacy Groups, Labor Unions, Religions)

Contributions Eagerly Sought

References

One-Page Link Guide to OSINT References - Covers history, context, process, practice, and tools

Business Week "The Power of Us"

Forbes "Reinventing Intelligence"

TIME "The New Craft of Intelligence"

NATO Open Source Intelligence Handbook (2001)

NATO Open Source Intelligence Reader (2002)

NATO Intelligence Exploitation of the Internet (2002)

[http:www./oss.net OSS.Net] - International public OSINT network and website, 30,000 pages,updated continuously.

Terms of Reference for Future of Intelligence - provides matrix for the war between secret and open intelligence proponents regarding funding, priorities, and methods.

Multinational Information Operations Center - Briefing on Moving to the Next Level

Information Operations: The Book - Briefing on Book that Takes IO to the Next Level

Annual IO/OSINT/PKI Conference - Annual Conference (since 1992) for Forward Thinkers (100 Scholarships)

IO: Putting the I Back Into Dime - Monograph from the Strategic Studies Institute of the US Army War College

The New Craft of Intelligence - Monograph from the Strategic Studies Institute of the US Army War College

Four Free Books Online ON INTEL, NEW CRAFT, IO, and PKI

External links (Vendors)

• BBC Monitoring Subscription service from the BBC.
• C4I.org- Private open-source intelligence clearinghouse.
• Dialog leader in providing online-based information services to organizations seeking competitive advantages.
• EastView Since 1989, East View has been the world's leader in providing high-quality information services from Russia and the NIS to the widest variety of customers.
• Factiva a Dow Jones & Reuters Company, provides world class global content, including Dow Jones and Reuters Newswires and The Wall Street Journal.
• Infosphere AB- Corporate OSINT from Commercial Intelligence & Knowledge Strategy consultancy Infosphere AB (Sweden).
• Lexis Nexis provides authoritative legal, news, public records and business information; including tax and regulatory publications in online, print or CD-ROM formats
• MarketResearch.com is the leading provider of global market intelligence products and services.
• Open Sources Center Subscription-based OSINT from ISRIA.
• The OSINT Center Geopolitical, Tourism and Energy OSINT from Alan Simpson
• SENTINEL- Government and private sector OSINT services from Elsag Solutions AG (Switzerland).
• Silobreaker- integrated sources and analytical webservice created by and for OSINT and CI users- Intelligence for Everyone (2005)
• World News Connection - global news from Federal Broadcast Information Service (FBIS is run by CIA, now under the DNI Open Source Center(OSC)).